2022-03-02T06:32:20.669 INFO:root:teuthology version: 1.1.0-cb5b1a75 2022-03-02T06:32:20.671 DEBUG:teuthology.run:Teuthology command: teuthology --name ozeneva-2022-03-01_17:32:13-rados-wip-omri-tracing-compiled-distro-basic-smithi --archive /home/teuthworker/archive/ozeneva-2022-03-01_17:32:13-rados-wip-omri-tracing-compiled-distro-basic-smithi/6714332 --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/flannel rook/1.7.2} --verbose --owner scheduled_ozeneva@teuthology -- /home/teuthworker/archive/ozeneva-2022-03-01_17:32:13-rados-wip-omri-tracing-compiled-distro-basic-smithi/6714332/orig.config.yaml 2022-03-02T06:32:20.706 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-02T06:32:20.843 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ozeneva-2022-03-01_17:32:13-rados-wip-omri-tracing-compiled-distro-basic-smithi/6714332 branch: wip-omri-tracing-compiled description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/flannel rook/1.7.2} email: null first_in_suite: false job_id: '6714332' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ozeneva-2022-03-01_17:32:13-rados-wip-omri-tracing-compiled-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-omri-tracing-compiled ceph: conf: global: osd crush chooseleaf type: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_branch: v1.7.2 rook_image: rook/ceph:v1.7.2 workunit: branch: wip-omri-tracing-compiled sha1: 8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 owner: scheduled_ozeneva@teuthology priority: 100 repo: git://git.ceph.com/ceph-ci.git roles: - - host.a - client.a sha1: 8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 sleep_before_teardown: 0 suite: rados suite_branch: wip-omri-tracing-compiled suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 targets: smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCy+SbMFlDwmyh3w0jPoL1vNr07mvmO0i8M21sAJuFQjCW3QyY9Ht9AzQej9H/lfLnM0t4oC3TTcWJ8oznUfwZBex0/m4/UwJdd1/niwFVAxII31E5D3co4+oZO+7NHjIGy7SBhD+1DMSlJ/C1vrkInUcVOfInT092CdZRn0v40bSDNCW9zrj+Re2M7u1fIfyb75wnWdfhIJEB/qHFT6PXeDfhBq56/Wio1NsPCwcyFvFpxPscizLNVpwWaRvuTNZf6FtNW+GvWkXgWMRua2eXMQArfwT/98omXwf1HgutO8MEpw4bFG09aqaSR96vkccwjIhFiIZ7Y2Tfn0DpNBcg5hvWnA+fTL081+ir7a9bMBi/wthJPgHs9/i19yIgCFqZmUZWEFAHlZ07PmIj9IAdECTL4ko5a58xDKSq5eMvfZCI8nO8DRWjGjnG8JFgCL9seN3qcVjhZHh3+78SOtgHuG3xkcfP7bCJmiCNxkq91vhTVD7f05QDiuDjPDuKZPzE= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - install: host.a: null - radosbench: clients: - client.a - rook.shell: commands: - 'ceph orch host label add `hostname` foo ceph orch host ls | grep foo ceph orch host label rm `hostname` foo ceph orch host ls | grep -v foo ' - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: cb5b1a75970b7aa81700a0ab081943b3146d40d0 timestamp: 2022-03-01_17:32:13 tube: smithi user: ozeneva verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-02T06:32:20.844 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa; will attempt to use it 2022-03-02T06:32:20.845 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks 2022-03-02T06:32:20.845 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-02T06:32:20.847 INFO:teuthology.task.internal:Checking packages... 2022-03-02T06:32:20.869 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2' 2022-03-02T06:32:20.869 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-02T06:32:20.870 INFO:teuthology.packaging:ref: None 2022-03-02T06:32:20.870 INFO:teuthology.packaging:tag: None 2022-03-02T06:32:20.871 INFO:teuthology.packaging:branch: wip-omri-tracing-compiled 2022-03-02T06:32:20.871 INFO:teuthology.packaging:sha1: 8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 2022-03-02T06:32:20.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-omri-tracing-compiled 2022-03-02T06:32:21.298 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-10916-g8b0a50f4-1focal 2022-03-02T06:32:21.299 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-02T06:32:21.311 INFO:teuthology.task.internal:no buildpackages task found 2022-03-02T06:32:21.311 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-02T06:32:21.430 INFO:teuthology.task.internal:Saving configuration 2022-03-02T06:32:21.443 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-02T06:32:21.559 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-02T06:32:21.585 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-01_17:32:13-rados-wip-omri-tracing-compiled-distro-basic-smithi/6714332', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-02 06:26:55.118993', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMAxPAPOH8ZJH/oLUQvsi1N4fRYWA4WpUTHAMcfX/8Sm'} 2022-03-02T06:32:21.586 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-02T06:32:21.740 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-02T06:32:21.741 INFO:teuthology.run_tasks:Running task console_log... 2022-03-02T06:32:21.910 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa2602ffe18>, signals=[15]) 2022-03-02T06:32:21.911 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-02T06:32:22.095 INFO:teuthology.task.internal:Opening connections... 2022-03-02T06:32:22.095 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2022-03-02T06:32:22.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-02T06:32:22.155 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-02T06:32:22.207 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2022-03-02T06:32:22.243 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2022-03-02T06:32:22.243 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2022-03-02T06:32:22.294 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Ubuntu" 2022-03-02T06:32:22.295 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-02T06:32:22.295 INFO:teuthology.orchestra.run.smithi136.stdout:ID=ubuntu 2022-03-02T06:32:22.295 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE=debian 2022-03-02T06:32:22.295 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-02T06:32:22.296 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="20.04" 2022-03-02T06:32:22.296 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-02T06:32:22.296 INFO:teuthology.orchestra.run.smithi136.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-02T06:32:22.296 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-02T06:32:22.297 INFO:teuthology.orchestra.run.smithi136.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-02T06:32:22.297 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_CODENAME=focal 2022-03-02T06:32:22.297 INFO:teuthology.orchestra.run.smithi136.stdout:UBUNTU_CODENAME=focal 2022-03-02T06:32:22.298 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2022-03-02T06:32:22.317 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-02T06:32:22.329 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-02T06:32:22.436 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-02T06:32:22.437 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2022-03-02T06:32:22.444 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-02T06:32:22.589 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-02T06:32:22.589 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2022-03-02T06:32:22.599 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-02T06:32:22.708 INFO:teuthology.run_tasks:Running task kernel... 2022-03-02T06:32:22.779 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-02T06:32:22.780 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-02T06:32:22.780 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}} 2022-03-02T06:32:22.781 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-02T06:32:22.781 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2022-03-02T06:32:22.798 INFO:teuthology.orchestra.run.smithi136.stdout:5.4.0-97-generic 2022-03-02T06:32:22.799 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.4.0-97-generic 2022-03-02T06:32:22.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2022-03-02T06:32:22.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2022-03-02T06:32:23.060 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-02T06:32:23.177 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-02T06:32:23.255 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-02T06:32:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-02T06:32:23.511 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-02T06:32:23.517 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-02T06:32:23.525 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-02T06:32:23.525 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-02T06:32:23.526 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-02T06:32:23.533 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-02T06:32:23.536 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-02T06:32:23.536 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-02T06:32:23.544 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-02T06:32:23.554 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-02T06:32:23.556 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-02T06:32:23.561 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-02T06:32:23.562 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-02T06:32:23.562 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-02T06:32:23.563 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-02T06:32:23.608 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-02T06:32:23.738 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-02T06:32:23.990 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-02T06:32:24.133 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-02T06:32:24.146 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-02T06:32:24.147 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-02T06:32:24.178 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-02T06:32:24.179 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-02T06:32:24.183 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-02T06:32:24.183 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [669 kB] 2022-03-02T06:32:24.209 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [905 kB] 2022-03-02T06:32:24.232 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-02T06:32:24.238 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-02T06:32:24.266 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-02T06:32:24.267 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-02T06:32:24.268 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-02T06:32:24.268 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-02T06:32:24.268 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.2 kB] 2022-03-02T06:32:24.296 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.4 kB] 2022-03-02T06:32:24.297 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.2 kB] 2022-03-02T06:32:24.378 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [732 B] 2022-03-02T06:32:26.580 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 9,913 kB in 2s (5,895 kB/s) 2022-03-02T06:32:27.760 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:32:27.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-02T06:32:27.856 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:32:28.029 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2022-03-02T06:32:28.030 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2022-03-02T06:32:28.170 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2022-03-02T06:32:28.171 INFO:teuthology.orchestra.run.smithi136.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-02T06:32:28.171 INFO:teuthology.orchestra.run.smithi136.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-02T06:32:28.172 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-02T06:32:28.172 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2022-03-02T06:32:28.172 INFO:teuthology.orchestra.run.smithi136.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-02T06:32:28.172 INFO:teuthology.orchestra.run.smithi136.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-02T06:32:28.214 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2022-03-02T06:32:28.215 INFO:teuthology.orchestra.run.smithi136.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-02T06:32:28.216 INFO:teuthology.orchestra.run.smithi136.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-02T06:32:28.216 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-02T06:32:28.216 INFO:teuthology.orchestra.run.smithi136.stdout: thermald upower usbmuxd 2022-03-02T06:32:28.397 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 13 newly installed, 0 to remove and 72 not upgraded. 2022-03-02T06:32:28.397 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 80.1 MB of archives. 2022-03-02T06:32:28.397 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-02T06:32:28.398 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-02T06:32:28.678 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-02T06:32:28.701 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-02T06:32:28.702 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-02T06:32:28.784 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-02T06:32:28.802 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-02T06:32:29.711 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-02T06:32:29.982 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-02T06:32:31.981 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-02T06:32:31.982 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-02T06:32:31.983 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-02T06:32:31.988 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-02T06:32:31.990 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-02T06:32:32.394 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 80.1 MB in 4s (21.2 MB/s) 2022-03-02T06:32:32.592 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-02T06:32:34.058 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-02T06:32:34.063 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-02T06:32:34.180 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-02T06:32:34.550 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-02T06:32:34.564 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-02T06:32:34.608 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-02T06:32:34.887 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-02T06:32:34.900 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-02T06:32:34.945 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-02T06:32:35.290 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-02T06:32:35.304 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-02T06:32:35.348 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-02T06:32:35.668 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-02T06:32:35.684 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-02T06:32:35.727 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-02T06:32:36.038 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-02T06:32:36.052 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-02T06:32:36.096 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-02T06:32:38.314 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-02T06:32:38.328 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-02T06:32:38.431 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-02T06:32:38.934 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-02T06:32:38.948 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-02T06:32:39.001 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-02T06:32:44.849 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-02T06:32:44.862 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-02T06:32:44.896 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-02T06:32:45.199 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-02T06:32:45.213 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-02T06:32:45.265 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-02T06:32:45.627 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package thermald. 2022-03-02T06:32:45.641 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-02T06:32:45.685 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-02T06:32:46.227 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package upower. 2022-03-02T06:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-02T06:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-02T06:32:46.701 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package usbmuxd. 2022-03-02T06:32:46.714 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-02T06:32:46.751 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-02T06:32:47.050 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-02T06:32:47.175 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-02T06:32:47.301 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-02T06:32:47.427 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-02T06:32:47.570 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-02T06:32:47.695 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-02T06:32:47.830 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-02T06:32:48.269 INFO:teuthology.orchestra.run.smithi136.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-02T06:32:48.316 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-02T06:32:48.490 INFO:teuthology.orchestra.run.smithi136.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-02T06:32:48.498 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-02T06:32:48.499 INFO:teuthology.orchestra.run.smithi136.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-02T06:32:48.957 INFO:teuthology.orchestra.run.smithi136.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-02T06:32:49.217 INFO:teuthology.orchestra.run.smithi136.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-02T06:32:49.245 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-02T06:32:49.538 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-02T06:32:49.538 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-02T06:32:50.027 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-02T06:32:51.263 INFO:teuthology.orchestra.run.smithi136.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-02T06:32:51.263 INFO:teuthology.orchestra.run.smithi136.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-02T06:32:51.426 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-02T06:32:52.769 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-02T06:32:54.079 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-02T06:32:54.213 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-02T06:32:57.752 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-02T06:32:58.430 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-02T06:32:58.960 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-02T06:32:59.864 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-02T06:32:59.864 INFO:teuthology.orchestra.run.smithi136.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-02T06:33:23.508 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-02T06:33:23.509 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub' 2022-03-02T06:33:23.513 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-02T06:33:23.536 INFO:teuthology.orchestra.run.smithi136.stdout:Generating grub configuration file ... 2022-03-02T06:33:23.842 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-02T06:33:23.867 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-02T06:33:24.257 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-02T06:33:24.268 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-02T06:33:26.869 INFO:teuthology.orchestra.run.smithi136.stdout:done 2022-03-02T06:33:27.220 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic-hwe-20.04 2022-03-02T06:33:27.264 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic-hwe-20.04 2022-03-02T06:33:27.264 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2022-03-02T06:33:27.264 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2022-03-02T06:33:27.265 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2022-03-02T06:33:27.265 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 19 2022-03-02T06:33:27.265 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2022-03-02T06:33:27.265 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2022-03-02T06:33:27.266 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta-hwe-5.13 2022-03-02T06:33:27.266 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-02T06:33:27.266 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-02T06:33:27.267 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-02T06:33:27.267 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2022-03-02T06:33:27.267 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2022-03-02T06:33:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2022-03-02T06:33:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2022-03-02T06:33:27.268 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-02T06:33:27.269 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-02T06:33:27.269 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-02T06:33:27.269 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-02T06:33:27.270 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2022-03-02T06:33:27.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2022-03-02T06:33:27.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2022-03-02T06:33:27.670 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-02T06:33:27.670 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-02T06:33:27.740 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-02T06:33:27.827 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-02T06:33:29.195 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:33:29.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-02T06:33:29.287 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:33:29.447 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2022-03-02T06:33:29.448 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2022-03-02T06:33:29.614 INFO:teuthology.orchestra.run.smithi136.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-02T06:33:29.615 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 0 newly installed, 0 to remove and 72 not upgraded. 2022-03-02T06:33:29.617 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic-hwe-20.04 2022-03-02T06:33:29.640 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic-hwe-20.04 2022-03-02T06:33:29.640 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2022-03-02T06:33:29.641 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2022-03-02T06:33:29.641 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2022-03-02T06:33:29.641 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 19 2022-03-02T06:33:29.642 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2022-03-02T06:33:29.642 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2022-03-02T06:33:29.642 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta-hwe-5.13 2022-03-02T06:33:29.643 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-02T06:33:29.643 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-02T06:33:29.643 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-02T06:33:29.643 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2022-03-02T06:33:29.644 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2022-03-02T06:33:29.644 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2022-03-02T06:33:29.644 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2022-03-02T06:33:29.645 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2022-03-02T06:33:29.651 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.QcwZg8kB2R 2022-03-02T06:33:29.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QcwZg8kB2R 2022-03-02T06:33:29.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.QcwZg8kB2R 2022-03-02T06:33:29.759 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.QcwZg8kB2R is 8KB 2022-03-02T06:33:29.774 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.QcwZg8kB2R 2022-03-02T06:33:29.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-02T06:33:29.837 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:33:29.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-02T06:33:29.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-02T06:33:29.874 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-02T06:33:29.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo update-grub 2022-03-02T06:33:31.110 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub' 2022-03-02T06:33:31.111 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-02T06:33:31.117 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2022-03-02T06:33:31.347 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-02T06:33:31.359 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-02T06:33:31.720 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-02T06:33:31.732 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-02T06:33:33.365 INFO:teuthology.orchestra.run.smithi136.stderr:done 2022-03-02T06:33:33.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2022-03-02T06:34:03.398 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-02T06:34:03.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2022-03-02T06:34:03.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-02T06:34:21.771 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2022-03-02T06:34:51.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2022-03-02T06:34:51.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-02T06:34:54.863 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2022-03-02T06:35:24.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2022-03-02T06:35:24.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-02T06:35:25.110 DEBUG:teuthology.orchestra.run.smithi136:> true 2022-03-02T06:35:25.612 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2022-03-02T06:35:25.612 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-30-generic"... 2022-03-02T06:35:25.613 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2022-03-02T06:35:25.618 INFO:teuthology.orchestra.run.smithi136.stdout:5.13.0-30-generic 2022-03-02T06:35:25.618 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-02T06:35:25.619 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-02T06:35:25.619 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-02T06:35:26.620 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-02T06:35:26.621 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-02T06:35:26.690 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2022-03-02T06:35:26.710 DEBUG:teuthology.parallel:result is None 2022-03-02T06:35:26.711 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-02T06:35:26.723 INFO:teuthology.task.internal:Creating test directory... 2022-03-02T06:35:26.725 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-02T06:35:26.733 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-02T06:35:26.868 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-02T06:35:26.923 INFO:teuthology.task.internal:Creating archive directory... 2022-03-02T06:35:26.924 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-02T06:35:26.949 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-02T06:35:27.150 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-02T06:35:27.151 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-02T06:35:27.181 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-02T06:35:27.189 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-02T06:35:27.191 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-02T06:35:27.346 INFO:teuthology.task.internal:Configuring sudo... 2022-03-02T06:35:27.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-02T06:35:27.363 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-02T06:35:27.460 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-02T06:35:27.460 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-02T06:35:27.468 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-02T06:35:27.474 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-02T06:35:27.479 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:35:27.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-02T06:35:27.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2022-03-02T06:35:27.591 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-02T06:35:27.688 INFO:teuthology.task.internal:Starting timer... 2022-03-02T06:35:27.688 INFO:teuthology.run_tasks:Running task pcp... 2022-03-02T06:35:27.834 INFO:teuthology.run_tasks:Running task selinux... 2022-03-02T06:35:27.961 DEBUG:teuthology.task.selinux:Excluding smithi136: OS 'ubuntu' does not support SELinux 2022-03-02T06:35:27.962 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-02T06:35:27.962 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-02T06:35:27.962 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-02T06:35:27.963 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-02T06:35:28.121 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-02T06:35:28.122 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-02T06:35:28.149 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-02T06:35:28.152 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-02T06:38:29.392 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2022-03-02T06:38:29.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2022-03-02T06:38:29.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-02T06:38:29.454 DEBUG:teuthology.orchestra.run.smithi136:> true 2022-03-02T06:38:29.576 INFO:teuthology.run_tasks:Running task clock... 2022-03-02T06:38:29.596 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-02T06:38:29.596 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-02T06:38:29.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-02T06:38:29.632 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-02T06:38:29.632 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: Command line: ntpd -gq 2022-03-02T06:38:29.633 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: proto: precision = 0.070 usec (-24) 2022-03-02T06:38:29.634 INFO:teuthology.orchestra.run.smithi136.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-02T06:38:29.634 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-02T06:38:29.635 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: restrict ::: KOD does nothing without LIMITED. 2022-03-02T06:38:29.635 INFO:teuthology.orchestra.run.smithi136.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-02T06:38:29.636 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: Listen and drop on 0 v6wildcard [::]:123 2022-03-02T06:38:29.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-02T06:38:29.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: Listen normally on 2 lo 127.0.0.1:123 2022-03-02T06:38:29.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: Listen normally on 3 enp3s0f1 172.21.15.136:123 2022-03-02T06:38:29.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: Listen normally on 4 lo [::1]:123 2022-03-02T06:38:29.638 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e43%5]:123 2022-03-02T06:38:29.638 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:29 ntpd[9035]: Listening on routing socket on fd #22 for interface updates 2022-03-02T06:38:30.634 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:30.634 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:30.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:30.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:30.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:30.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:30.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:30.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:30.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:30.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:30.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:30.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:30 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:32.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:32.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:32.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:32.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:32.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:32.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:32.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:32.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:32.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:32.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:32.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:32.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:32.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:32.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:32.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:32.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:32.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:32.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:32.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:32.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:32.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:32.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:32.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:32.641 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:32 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:34.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:34.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:34.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:34.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:34.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:34.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:34.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:34.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:34.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:34.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:34.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:34.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:34.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:34.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:34.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:34.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:34.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:34.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:34.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:34.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:34.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:34.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:34.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:34.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:34 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:36.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:36.635 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:36.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:36.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:36.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:36.636 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:36.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:36.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:36.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:36.637 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:36.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:36.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:36.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:36.638 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:36.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:36.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:36.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-02T06:38:36.639 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220302: Permission denied 2022-03-02T06:38:36.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-02T06:38:36.640 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220302: Permission denied 2022-03-02T06:38:36.640 INFO:teuthology.orchestra.run.smithi136.stdout: 2 Mar 06:38:36 ntpd[9035]: ntpd: time slew -0.000193 s 2022-03-02T06:38:36.641 INFO:teuthology.orchestra.run.smithi136.stdout:ntpd: time slew -0.000193s 2022-03-02T06:38:36.641 INFO:teuthology.orchestra.run.smithi136.stderr: 2 Mar 06:38:36 ntpd[9035]: can't open /var/log/ntpstats/loopstats.20220302: Permission denied 2022-03-02T06:38:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2022-03-02T06:38:36.707 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2022-03-02T06:38:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-02T06:38:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-02T06:38:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-02T06:38:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-02T06:38:36.709 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-02T06:38:36.724 INFO:tasks.kubeadm:Kubeadm start 2022-03-02T06:38:36.724 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-03-02T06:38:36.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-02T06:38:36.753 INFO:teuthology.orchestra.run.smithi136.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-02T06:38:36.762 INFO:teuthology.orchestra.run.smithi136.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-02T06:38:36.771 INFO:teuthology.orchestra.run.smithi136.stdout:net.ipv4.ip_forward = 1 2022-03-02T06:38:36.822 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:38:36.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-02T06:38:36.838 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:38:36.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/docker/daemon.json 2022-03-02T06:38:36.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl restart docker || true 2022-03-02T06:38:37.967 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-02T06:38:37.974 INFO:tasks.kubeadm:172.21.15.136 is in 172.21.0.0/20 at pos 3975 2022-03-02T06:38:37.990 INFO:tasks.kubeadm:Pod subnet: 10.252.56.0/21 2022-03-02T06:38:37.990 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-02T06:38:37.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-02T06:38:38.013 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:38:38.014 INFO:teuthology.orchestra.run.smithi136.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-02T06:38:38.014 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:38:38.257 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-02T06:38:38.258 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-02T06:38:38.329 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-02T06:38:38.412 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-02T06:38:39.746 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:38:39.914 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2022-03-02T06:38:39.915 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2022-03-02T06:38:40.003 INFO:teuthology.orchestra.run.smithi136.stdout:65 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-02T06:38:40.016 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:38:40.017 INFO:teuthology.orchestra.run.smithi136.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-02T06:38:40.017 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:38:40.071 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:38:40.243 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2022-03-02T06:38:40.244 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2022-03-02T06:38:40.412 INFO:teuthology.orchestra.run.smithi136.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-02T06:38:40.413 INFO:teuthology.orchestra.run.smithi136.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-02T06:38:40.413 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2022-03-02T06:38:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: apt-transport-https 2022-03-02T06:38:40.609 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 1 newly installed, 0 to remove and 65 not upgraded. 2022-03-02T06:38:40.610 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 4,680 B of archives. 2022-03-02T06:38:40.610 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-02T06:38:40.611 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-02T06:38:40.839 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: unable to initialize frontend: Dialog 2022-03-02T06:38:40.840 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-02T06:38:40.840 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: falling back to frontend: Readline 2022-03-02T06:38:40.860 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: unable to initialize frontend: Readline 2022-03-02T06:38:40.860 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-02T06:38:40.860 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: falling back to frontend: Teletype 2022-03-02T06:38:40.865 INFO:teuthology.orchestra.run.smithi136.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-02T06:38:40.916 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 4,680 B in 0s (24.9 kB/s) 2022-03-02T06:38:41.005 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package apt-transport-https. 2022-03-02T06:38:41.048 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-02T06:38:41.052 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-02T06:38:41.111 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-02T06:38:41.406 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-02T06:38:41.926 INFO:teuthology.orchestra.run.smithi136.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-02T06:38:41.936 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:38:41.937 INFO:teuthology.orchestra.run.smithi136.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-02T06:38:41.937 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:38:42.061 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-02T06:38:42.176 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-02T06:38:42.258 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-02T06:38:42.340 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-02T06:38:42.534 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-02T06:38:42.727 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-02T06:38:42.862 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 63.3 kB in 1s (81.1 kB/s) 2022-03-02T06:38:43.713 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:38:43.885 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2022-03-02T06:38:43.886 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2022-03-02T06:38:43.974 INFO:teuthology.orchestra.run.smithi136.stdout:65 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-02T06:38:43.986 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:38:43.986 INFO:teuthology.orchestra.run.smithi136.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-02T06:38:43.986 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:38:44.040 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:38:44.198 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2022-03-02T06:38:44.202 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2022-03-02T06:38:44.328 INFO:teuthology.orchestra.run.smithi136.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-02T06:38:44.329 INFO:teuthology.orchestra.run.smithi136.stdout:bridge-utils set to manually installed. 2022-03-02T06:38:44.329 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2022-03-02T06:38:44.330 INFO:teuthology.orchestra.run.smithi136.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-02T06:38:44.330 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2022-03-02T06:38:44.330 INFO:teuthology.orchestra.run.smithi136.stdout: nftables 2022-03-02T06:38:44.378 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2022-03-02T06:38:44.379 INFO:teuthology.orchestra.run.smithi136.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-02T06:38:44.573 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 8 newly installed, 0 to remove and 65 not upgraded. 2022-03-02T06:38:44.573 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 73.6 MB of archives. 2022-03-02T06:38:44.573 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-02T06:38:44.574 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-02T06:38:44.602 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-02T06:38:44.756 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-02T06:38:44.838 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-02T06:38:44.867 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-02T06:38:45.353 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-02T06:38:45.759 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-02T06:38:45.873 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-02T06:38:46.243 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: unable to initialize frontend: Dialog 2022-03-02T06:38:46.243 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-02T06:38:46.244 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: falling back to frontend: Readline 2022-03-02T06:38:46.251 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: unable to initialize frontend: Readline 2022-03-02T06:38:46.251 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-02T06:38:46.251 INFO:teuthology.orchestra.run.smithi136.stderr:debconf: falling back to frontend: Teletype 2022-03-02T06:38:46.257 INFO:teuthology.orchestra.run.smithi136.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-02T06:38:46.305 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 73.6 MB in 2s (45.0 MB/s) 2022-03-02T06:38:46.385 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package conntrack. 2022-03-02T06:38:46.428 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-02T06:38:46.431 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-02T06:38:46.467 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-02T06:38:46.770 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cri-tools. 2022-03-02T06:38:46.782 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-02T06:38:46.820 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-02T06:38:47.931 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ebtables. 2022-03-02T06:38:47.946 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-02T06:38:48.030 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-02T06:38:48.293 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-02T06:38:48.308 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-02T06:38:48.376 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-02T06:38:49.837 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package socat. 2022-03-02T06:38:49.849 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-02T06:38:49.887 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-02T06:38:50.290 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package kubelet. 2022-03-02T06:38:50.303 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-02T06:38:50.340 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-02T06:38:52.792 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package kubectl. 2022-03-02T06:38:52.807 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-02T06:38:52.850 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-02T06:38:54.061 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package kubeadm. 2022-03-02T06:38:54.075 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-02T06:38:54.127 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-02T06:38:55.301 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-02T06:38:55.426 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-02T06:38:55.553 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-02T06:38:56.423 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-02T06:38:56.551 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-02T06:38:56.677 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-02T06:38:56.805 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-02T06:38:57.025 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-02T06:38:57.540 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-02T06:38:58.034 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-02T06:38:58.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-02T06:39:12.638 INFO:teuthology.orchestra.run.smithi136.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-02T06:39:15.548 INFO:teuthology.orchestra.run.smithi136.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-02T06:39:17.726 INFO:teuthology.orchestra.run.smithi136.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-02T06:39:22.044 INFO:teuthology.orchestra.run.smithi136.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-02T06:39:23.510 INFO:teuthology.orchestra.run.smithi136.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-02T06:39:28.877 INFO:teuthology.orchestra.run.smithi136.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-02T06:39:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-02T06:39:31.256 INFO:tasks.kubeadm:Token: abcdef.4ms8v4f2onuxxp53 2022-03-02T06:39:31.256 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): '172.21.15.136'} 2022-03-02T06:39:31.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo kubeadm init --node-name smithi136 --token abcdef.4ms8v4f2onuxxp53 --pod-network-cidr 10.252.56.0/21 2022-03-02T06:39:31.469 INFO:teuthology.orchestra.run.smithi136.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-02T06:39:31.470 INFO:teuthology.orchestra.run.smithi136.stdout:[preflight] Running pre-flight checks 2022-03-02T06:39:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-02T06:39:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-02T06:39:31.757 INFO:teuthology.orchestra.run.smithi136.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-02T06:39:32.083 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-02T06:39:32.161 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "ca" certificate and key 2022-03-02T06:39:32.384 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "apiserver" certificate and key 2022-03-02T06:39:32.385 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi136] and IPs [10.96.0.1 172.21.15.136] 2022-03-02T06:39:32.568 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-02T06:39:32.711 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-02T06:39:32.805 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-02T06:39:32.957 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-02T06:39:33.050 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-02T06:39:33.051 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi136] and IPs [172.21.15.136 127.0.0.1 ::1] 2022-03-02T06:39:33.403 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-02T06:39:33.404 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi136] and IPs [172.21.15.136 127.0.0.1 ::1] 2022-03-02T06:39:33.518 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-02T06:39:33.662 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-02T06:39:33.736 INFO:teuthology.orchestra.run.smithi136.stdout:[certs] Generating "sa" key and public key 2022-03-02T06:39:33.737 INFO:teuthology.orchestra.run.smithi136.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-02T06:39:33.877 INFO:teuthology.orchestra.run.smithi136.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-02T06:39:34.144 INFO:teuthology.orchestra.run.smithi136.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-02T06:39:34.278 INFO:teuthology.orchestra.run.smithi136.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-02T06:39:34.466 INFO:teuthology.orchestra.run.smithi136.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-02T06:39:34.480 INFO:teuthology.orchestra.run.smithi136.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-02T06:39:34.482 INFO:teuthology.orchestra.run.smithi136.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-02T06:39:34.482 INFO:teuthology.orchestra.run.smithi136.stdout:[kubelet-start] Starting the kubelet 2022-03-02T06:39:34.713 INFO:teuthology.orchestra.run.smithi136.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-02T06:39:34.713 INFO:teuthology.orchestra.run.smithi136.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-02T06:39:34.717 INFO:teuthology.orchestra.run.smithi136.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-02T06:39:34.718 INFO:teuthology.orchestra.run.smithi136.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-02T06:39:34.719 INFO:teuthology.orchestra.run.smithi136.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-02T06:39:34.722 INFO:teuthology.orchestra.run.smithi136.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-02T06:39:52.224 INFO:teuthology.orchestra.run.smithi136.stdout:[apiclient] All control plane components are healthy after 17.502651 seconds 2022-03-02T06:39:52.225 INFO:teuthology.orchestra.run.smithi136.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-02T06:39:52.248 INFO:teuthology.orchestra.run.smithi136.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-02T06:39:52.248 INFO:teuthology.orchestra.run.smithi136.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-02T06:39:52.801 INFO:teuthology.orchestra.run.smithi136.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-02T06:39:52.801 INFO:teuthology.orchestra.run.smithi136.stdout:[mark-control-plane] Marking the node smithi136 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-02T06:39:52.802 INFO:teuthology.orchestra.run.smithi136.stdout:[mark-control-plane] Marking the node smithi136 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-02T06:39:53.314 INFO:teuthology.orchestra.run.smithi136.stdout:[bootstrap-token] Using token: abcdef.4ms8v4f2onuxxp53 2022-03-02T06:39:53.314 INFO:teuthology.orchestra.run.smithi136.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-02T06:39:53.344 INFO:teuthology.orchestra.run.smithi136.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-02T06:39:53.417 INFO:teuthology.orchestra.run.smithi136.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-02T06:39:53.426 INFO:teuthology.orchestra.run.smithi136.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-02T06:39:53.433 INFO:teuthology.orchestra.run.smithi136.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-02T06:39:53.438 INFO:teuthology.orchestra.run.smithi136.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-02T06:39:53.481 INFO:teuthology.orchestra.run.smithi136.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-02T06:39:53.923 INFO:teuthology.orchestra.run.smithi136.stdout:[addons] Applied essential addon: CoreDNS 2022-03-02T06:39:54.047 INFO:teuthology.orchestra.run.smithi136.stdout:[addons] Applied essential addon: kube-proxy 2022-03-02T06:39:54.048 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:39:54.048 INFO:teuthology.orchestra.run.smithi136.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-02T06:39:54.048 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:39:54.048 INFO:teuthology.orchestra.run.smithi136.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-02T06:39:54.049 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:39:54.049 INFO:teuthology.orchestra.run.smithi136.stdout: mkdir -p $HOME/.kube 2022-03-02T06:39:54.049 INFO:teuthology.orchestra.run.smithi136.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-02T06:39:54.050 INFO:teuthology.orchestra.run.smithi136.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-02T06:39:54.050 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:39:54.050 INFO:teuthology.orchestra.run.smithi136.stdout:Alternatively, if you are the root user, you can run: 2022-03-02T06:39:54.051 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:39:54.052 INFO:teuthology.orchestra.run.smithi136.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-02T06:39:54.052 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:39:54.053 INFO:teuthology.orchestra.run.smithi136.stdout:You should now deploy a pod network to the cluster. 2022-03-02T06:39:54.053 INFO:teuthology.orchestra.run.smithi136.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-02T06:39:54.053 INFO:teuthology.orchestra.run.smithi136.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-02T06:39:54.053 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:39:54.054 INFO:teuthology.orchestra.run.smithi136.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-02T06:39:54.054 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:39:54.054 INFO:teuthology.orchestra.run.smithi136.stdout:kubeadm join 172.21.15.136:6443 --token abcdef.4ms8v4f2onuxxp53 \ 2022-03-02T06:39:54.054 INFO:teuthology.orchestra.run.smithi136.stdout: --discovery-token-ca-cert-hash sha256:8d7cdc07680fe37039bb1ecb8f78aa0967ee1484e2c339e4e9bbe6449179fd21 2022-03-02T06:39:54.056 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:39:54.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-02T06:39:54.071 INFO:tasks.kubeadm:Setting up kubectl 2022-03-02T06:39:54.072 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-02T06:39:54.132 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:39:54.132 DEBUG:teuthology.orchestra.run.smithi136:> dd of=.kube/config 2022-03-02T06:39:54.138 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:39:54.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/root/.kube/config 2022-03-02T06:39:54.150 DEBUG:teuthology.orchestra.run.smithi136:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-03-02T06:39:54.207 INFO:teuthology.orchestra.run.smithi136.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-03-02T06:39:54.207 INFO:teuthology.orchestra.run.smithi136.stderr: Dload Upload Total Spent Left Speed 2022-03-02T06:39:54.256 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2022-03-02T06:39:54.257 INFO:teuthology.orchestra.run.smithi136.stdout:apiVersion: policy/v1beta1 2022-03-02T06:39:54.257 INFO:teuthology.orchestra.run.smithi136.stdout:kind: PodSecurityPolicy 2022-03-02T06:39:54.257 INFO:teuthology.orchestra.run.smithi136.stdout:metadata: 2022-03-02T06:39:54.258 INFO:teuthology.orchestra.run.smithi136.stdout: name: psp.flannel.unprivileged 2022-03-02T06:39:54.258 INFO:teuthology.orchestra.run.smithi136.stdout: annotations: 2022-03-02T06:39:54.258 INFO:teuthology.orchestra.run.smithi136.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-03-02T06:39:54.258 INFO:teuthology.orchestra.run.smithi136.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-03-02T06:39:54.258 INFO:teuthology.orchestra.run.smithi136.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-03-02T06:39:54.259 INFO:teuthology.orchestra.run.smithi136.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-03-02T06:39:54.259 INFO:teuthology.orchestra.run.smithi136.stdout:spec: 2022-03-02T06:39:54.259 INFO:teuthology.orchestra.run.smithi136.stdout: privileged: false 2022-03-02T06:39:54.259 INFO:teuthology.orchestra.run.smithi136.stdout: volumes: 2022-03-02T06:39:54.260 INFO:teuthology.orchestra.run.smithi136.stdout: - configMap 2022-03-02T06:39:54.260 INFO:teuthology.orchestra.run.smithi136.stdout: - secret 2022-03-02T06:39:54.260 INFO:teuthology.orchestra.run.smithi136.stdout: - emptyDir 2022-03-02T06:39:54.260 INFO:teuthology.orchestra.run.smithi136.stdout: - hostPath 2022-03-02T06:39:54.261 INFO:teuthology.orchestra.run.smithi136.stdout: allowedHostPaths: 2022-03-02T06:39:54.261 INFO:teuthology.orchestra.run.smithi136.stdout: - pathPrefix: "/etc/cni/net.d" 2022-03-02T06:39:54.261 INFO:teuthology.orchestra.run.smithi136.stdout: - pathPrefix: "/etc/kube-flannel" 2022-03-02T06:39:54.261 INFO:teuthology.orchestra.run.smithi136.stdout: - pathPrefix: "/run/flannel" 2022-03-02T06:39:54.261 INFO:teuthology.orchestra.run.smithi136.stdout: readOnlyRootFilesystem: false 2022-03-02T06:39:54.262 INFO:teuthology.orchestra.run.smithi136.stdout: # Users and groups 2022-03-02T06:39:54.262 INFO:teuthology.orchestra.run.smithi136.stdout: runAsUser: 2022-03-02T06:39:54.262 INFO:teuthology.orchestra.run.smithi136.stdout: rule: RunAsAny 2022-03-02T06:39:54.262 INFO:teuthology.orchestra.run.smithi136.stdout: supplementalGroups: 2022-03-02T06:39:54.263 INFO:teuthology.orchestra.run.smithi136.stdout: rule: RunAsAny 2022-03-02T06:39:54.263 INFO:teuthology.orchestra.run.smithi136.stdout: fsGroup: 2022-03-02T06:39:54.263 INFO:teuthology.orchestra.run.smithi136.stdout: rule: RunAsAny 2022-03-02T06:39:54.263 INFO:teuthology.orchestra.run.smithi136.stdout: # Privilege Escalation 2022-03-02T06:39:54.264 INFO:teuthology.orchestra.run.smithi136.stdout: allowPrivilegeEscalation: false 2022-03-02T06:39:54.264 INFO:teuthology.orchestra.run.smithi136.stdout: defaultAllowPrivilegeEscalation: false 2022-03-02T06:39:54.264 INFO:teuthology.orchestra.run.smithi136.stdout: # Capabilities 2022-03-02T06:39:54.264 INFO:teuthology.orchestra.run.smithi136.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-03-02T06:39:54.264 INFO:teuthology.orchestra.run.smithi136.stdout: defaultAddCapabilities: [] 2022-03-02T06:39:54.265 INFO:teuthology.orchestra.run.smithi136.stdout: requiredDropCapabilities: [] 2022-03-02T06:39:54.265 INFO:teuthology.orchestra.run.smithi136.stdout: # Host namespaces 2022-03-02T06:39:54.265 INFO:teuthology.orchestra.run.smithi136.stdout: hostPID: false 2022-03-02T06:39:54.265 INFO:teuthology.orchestra.run.smithi136.stdout: hostIPC: false 2022-03-02T06:39:54.266 INFO:teuthology.orchestra.run.smithi136.stdout: hostNetwork: true 2022-03-02T06:39:54.266 INFO:teuthology.orchestra.run.smithi136.stdout: hostPorts: 2022-03-02T06:39:54.266 INFO:teuthology.orchestra.run.smithi136.stdout: - min: 0 2022-03-02T06:39:54.266 INFO:teuthology.orchestra.run.smithi136.stdout: max: 65535 2022-03-02T06:39:54.267 INFO:teuthology.orchestra.run.smithi136.stdout: # SELinux 2022-03-02T06:39:54.267 INFO:teuthology.orchestra.run.smithi136.stdout: seLinux: 2022-03-02T06:39:54.267 INFO:teuthology.orchestra.run.smithi136.stdout: # SELinux is unused in CaaSP 2022-03-02T06:39:54.267 INFO:teuthology.orchestra.run.smithi136.stdout: rule: 'RunAsAny' 2022-03-02T06:39:54.268 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2022-03-02T06:39:54.268 INFO:teuthology.orchestra.run.smithi136.stdout:kind: ClusterRole 2022-03-02T06:39:54.268 INFO:teuthology.orchestra.run.smithi136.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-02T06:39:54.268 INFO:teuthology.orchestra.run.smithi136.stdout:metadata: 2022-03-02T06:39:54.268 INFO:teuthology.orchestra.run.smithi136.stdout: name: flannel 2022-03-02T06:39:54.269 INFO:teuthology.orchestra.run.smithi136.stdout:rules: 2022-03-02T06:39:54.269 INFO:teuthology.orchestra.run.smithi136.stdout:- apiGroups: ['extensions'] 2022-03-02T06:39:54.269 INFO:teuthology.orchestra.run.smithi136.stdout: resources: ['podsecuritypolicies'] 2022-03-02T06:39:54.269 INFO:teuthology.orchestra.run.smithi136.stdout: verbs: ['use'] 2022-03-02T06:39:54.270 INFO:teuthology.orchestra.run.smithi136.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-03-02T06:39:54.270 INFO:teuthology.orchestra.run.smithi136.stdout:- apiGroups: 2022-03-02T06:39:54.270 INFO:teuthology.orchestra.run.smithi136.stdout: - "" 2022-03-02T06:39:54.271 INFO:teuthology.orchestra.run.smithi136.stdout: resources: 2022-03-02T06:39:54.271 INFO:teuthology.orchestra.run.smithi136.stdout: - pods 2022-03-02T06:39:54.271 INFO:teuthology.orchestra.run.smithi136.stdout: verbs: 2022-03-02T06:39:54.271 INFO:teuthology.orchestra.run.smithi136.stdout: - get 2022-03-02T06:39:54.271 INFO:teuthology.orchestra.run.smithi136.stdout:- apiGroups: 2022-03-02T06:39:54.272 INFO:teuthology.orchestra.run.smithi136.stdout: - "" 2022-03-02T06:39:54.272 INFO:teuthology.orchestra.run.smithi136.stdout: resources: 2022-03-02T06:39:54.272 INFO:teuthology.orchestra.run.smithi136.stdout: - nodes 2022-03-02T06:39:54.272 INFO:teuthology.orchestra.run.smithi136.stdout: verbs: 2022-03-02T06:39:54.273 INFO:teuthology.orchestra.run.smithi136.stdout: - list 2022-03-02T06:39:54.273 INFO:teuthology.orchestra.run.smithi136.stdout: - watch 2022-03-02T06:39:54.273 INFO:teuthology.orchestra.run.smithi136.stdout:- apiGroups: 2022-03-02T06:39:54.273 INFO:teuthology.orchestra.run.smithi136.stdout: - "" 2022-03-02T06:39:54.274 INFO:teuthology.orchestra.run.smithi136.stdout: resources: 2022-03-02T06:39:54.274 INFO:teuthology.orchestra.run.smithi136.stdout: - nodes/status 2022-03-02T06:39:54.274 INFO:teuthology.orchestra.run.smithi136.stdout: verbs: 2022-03-02T06:39:54.274 INFO:teuthology.orchestra.run.smithi136.stdout: - patch 2022-03-02T06:39:54.274 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2022-03-02T06:39:54.275 INFO:teuthology.orchestra.run.smithi136.stdout:kind: ClusterRoleBinding 2022-03-02T06:39:54.275 INFO:teuthology.orchestra.run.smithi136.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-02T06:39:54.275 INFO:teuthology.orchestra.run.smithi136.stdout:metadata: 2022-03-02T06:39:54.275 INFO:teuthology.orchestra.run.smithi136.stdout: name: flannel 2022-03-02T06:39:54.276 INFO:teuthology.orchestra.run.smithi136.stdout:roleRef: 2022-03-02T06:39:54.277 INFO:teuthology.orchestra.run.smithi136.stdout: apiGroup: rbac.authorization.k8s.io 2022-03-02T06:39:54.277 INFO:teuthology.orchestra.run.smithi136.stdout: kind: ClusterRole 2022-03-02T06:39:54.277 INFO:teuthology.orchestra.run.smithi136.stdout: name: flannel 2022-03-02T06:39:54.278 INFO:teuthology.orchestra.run.smithi136.stdout:subjects: 2022-03-02T06:39:54.278 INFO:teuthology.orchestra.run.smithi136.stdout:- kind: ServiceAccount 2022-03-02T06:39:54.278 INFO:teuthology.orchestra.run.smithi136.stdout: name: flannel 2022-03-02T06:39:54.279 INFO:teuthology.orchestra.run.smithi136.stdout: namespace: kube-system 2022-03-02T06:39:54.279 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2022-03-02T06:39:54.279 INFO:teuthology.orchestra.run.smithi136.stdout:apiVersion: v1 2022-03-02T06:39:54.280 INFO:teuthology.orchestra.run.smithi136.stdout:kind: ServiceAccount 2022-03-02T06:39:54.280 INFO:teuthology.orchestra.run.smithi136.stdout:metadata: 2022-03-02T06:39:54.281 INFO:teuthology.orchestra.run.smithi136.stdout: name: flannel 2022-03-02T06:39:54.281 INFO:teuthology.orchestra.run.smithi136.stdout: namespace: kube-system 2022-03-02T06:39:54.282 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2022-03-02T06:39:54.282 INFO:teuthology.orchestra.run.smithi136.stdout:kind: ConfigMap 2022-03-02T06:39:54.283 INFO:teuthology.orchestra.run.smithi136.stdout:apiVersion: v1 2022-03-02T06:39:54.283 INFO:teuthology.orchestra.run.smithi136.stdout:metadata: 2022-03-02T06:39:54.283 INFO:teuthology.orchestra.run.smithi136.stdout: name: kube-flannel-cfg 2022-03-02T06:39:54.284 INFO:teuthology.orchestra.run.smithi136.stdout: namespace: kube-system 2022-03-02T06:39:54.284 INFO:teuthology.orchestra.run.smithi136.stdout: labels: 2022-03-02T06:39:54.285 INFO:teuthology.orchestra.run.smithi136.stdout: tier: node 2022-03-02T06:39:54.285 INFO:teuthology.orchestra.run.smithi136.stdout: app: flannel 2022-03-02T06:39:54.286 INFO:teuthology.orchestra.run.smithi136.stdout:data: 2022-03-02T06:39:54.286 INFO:teuthology.orchestra.run.smithi136.stdout: cni-conf.json: | 2022-03-02T06:39:54.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2022-03-02T06:39:54.287 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "cbr0", 2022-03-02T06:39:54.287 INFO:teuthology.orchestra.run.smithi136.stdout: "cniVersion": "0.3.1", 2022-03-02T06:39:54.288 INFO:teuthology.orchestra.run.smithi136.stdout: "plugins": [ 2022-03-02T06:39:54.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2022-03-02T06:39:54.289 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "flannel", 2022-03-02T06:39:54.289 INFO:teuthology.orchestra.run.smithi136.stdout: "delegate": { 2022-03-02T06:39:54.289 INFO:teuthology.orchestra.run.smithi136.stdout: "hairpinMode": true, 2022-03-02T06:39:54.290 INFO:teuthology.orchestra.run.smithi136.stdout: "isDefaultGateway": true 2022-03-02T06:39:54.290 INFO:teuthology.orchestra.run.smithi136.stdout: } 2022-03-02T06:39:54.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2022-03-02T06:39:54.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2022-03-02T06:39:54.292 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "portmap", 2022-03-02T06:39:54.292 INFO:teuthology.orchestra.run.smithi136.stdout: "capabilities": { 2022-03-02T06:39:54.292 INFO:teuthology.orchestra.run.smithi136.stdout: "portMappings": true 2022-03-02T06:39:54.293 INFO:teuthology.orchestra.run.smithi136.stdout: } 2022-03-02T06:39:54.293 INFO:teuthology.orchestra.run.smithi136.stdout: } 2022-03-02T06:39:54.293 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2022-03-02T06:39:54.293 INFO:teuthology.orchestra.run.smithi136.stdout: } 2022-03-02T06:39:54.294 INFO:teuthology.orchestra.run.smithi136.stdout: net-conf.json: | 2022-03-02T06:39:54.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2022-03-02T06:39:54.294 INFO:teuthology.orchestra.run.smithi136.stdout: "Network": "10.244.0.0/16", 2022-03-02T06:39:54.294 INFO:teuthology.orchestra.run.smithi136.stdout: "Backend": { 2022-03-02T06:39:54.295 INFO:teuthology.orchestra.run.smithi136.stdout: "Type": "vxlan" 2022-03-02T06:39:54.295 INFO:teuthology.orchestra.run.smithi136.stdout: } 2022-03-02T06:39:54.295 INFO:teuthology.orchestra.run.smithi136.stdout: } 2022-03-02T06:39:54.295 INFO:teuthology.orchestra.run.smithi136.stdout:--- 2022-03-02T06:39:54.296 INFO:teuthology.orchestra.run.smithi136.stdout:apiVersion: apps/v1 2022-03-02T06:39:54.296 INFO:teuthology.orchestra.run.smithi136.stdout:kind: DaemonSet 2022-03-02T06:39:54.296 INFO:teuthology.orchestra.run.smithi136.stdout:metadata: 2022-03-02T06:39:54.296 INFO:teuthology.orchestra.run.smithi136.stdout: name: kube-flannel-ds 2022-03-02T06:39:54.296 INFO:teuthology.orchestra.run.smithi136.stdout: namespace: kube-system 2022-03-02T06:39:54.297 INFO:teuthology.orchestra.run.smithi136.stdout: labels: 2022-03-02T06:39:54.297 INFO:teuthology.orchestra.run.smithi136.stdout: tier: node 2022-03-02T06:39:54.297 INFO:teuthology.orchestra.run.smithi136.stdout: app: flannel 2022-03-02T06:39:54.297 INFO:teuthology.orchestra.run.smithi136.stdout:spec: 2022-03-02T06:39:54.298 INFO:teuthology.orchestra.run.smithi136.stdout: selector: 2022-03-02T06:39:54.298 INFO:teuthology.orchestra.run.smithi136.stdout: matchLabels: 2022-03-02T06:39:54.298 INFO:teuthology.orchestra.run.smithi136.stdout: app: flannel 2022-03-02T06:39:54.298 INFO:teuthology.orchestra.run.smithi136.stdout: template: 2022-03-02T06:39:54.299 INFO:teuthology.orchestra.run.smithi136.stdout: metadata: 2022-03-02T06:39:54.299 INFO:teuthology.orchestra.run.smithi136.stdout: labels: 2022-03-02T06:39:54.299 INFO:teuthology.orchestra.run.smithi136.stdout: tier: node 2022-03-02T06:39:54.299 INFO:teuthology.orchestra.run.smithi136.stdout: app: flannel 2022-03-02T06:39:54.300 INFO:teuthology.orchestra.run.smithi136.stdout: spec: 2022-03-02T06:39:54.300 INFO:teuthology.orchestra.run.smithi136.stdout: affinity: 2022-03-02T06:39:54.300 INFO:teuthology.orchestra.run.smithi136.stdout: nodeAffinity: 2022-03-02T06:39:54.300 INFO:teuthology.orchestra.run.smithi136.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-03-02T06:39:54.301 INFO:teuthology.orchestra.run.smithi136.stdout: nodeSelectorTerms: 2022-03-02T06:39:54.301 INFO:teuthology.orchestra.run.smithi136.stdout: - matchExpressions: 2022-03-02T06:39:54.301 INFO:teuthology.orchestra.run.smithi136.stdout: - key: kubernetes.io/os 2022-03-02T06:39:54.301 INFO:teuthology.orchestra.run.smithi136.stdout: operator: In 2022-03-02T06:39:54.302 INFO:teuthology.orchestra.run.smithi136.stdout: values: 2022-03-02T06:39:54.302 INFO:teuthology.orchestra.run.smithi136.stdout: - linux 2022-03-02T06:39:54.302 INFO:teuthology.orchestra.run.smithi136.stdout: hostNetwork: true 2022-03-02T06:39:54.302 INFO:teuthology.orchestra.run.smithi136.stdout: priorityClassName: system-node-critical 2022-03-02T06:39:54.303 INFO:teuthology.orchestra.run.smithi136.stdout: tolerations: 2022-03-02T06:39:54.303 INFO:teuthology.orchestra.run.smithi136.stdout: - operator: Exists 2022-03-02T06:39:54.303 INFO:teuthology.orchestra.run.smithi136.stdout: effect: NoSchedule 2022-03-02T06:39:54.303 INFO:teuthology.orchestra.run.smithi136.stdout: serviceAccountName: flannel 2022-03-02T06:39:54.304 INFO:teuthology.orchestra.run.smithi136.stdout: initContainers: 2022-03-02T06:39:54.304 INFO:teuthology.orchestra.run.smithi136.stdout: - name: install-cni-plugin 2022-03-02T06:39:54.304 INFO:teuthology.orchestra.run.smithi136.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-02T06:39:54.304 INFO:teuthology.orchestra.run.smithi136.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-03-02T06:39:54.305 INFO:teuthology.orchestra.run.smithi136.stdout: command: 2022-03-02T06:39:54.305 INFO:teuthology.orchestra.run.smithi136.stdout: - cp 2022-03-02T06:39:54.305 INFO:teuthology.orchestra.run.smithi136.stdout: args: 2022-03-02T06:39:54.306 INFO:teuthology.orchestra.run.smithi136.stdout: - -f 2022-03-02T06:39:54.306 INFO:teuthology.orchestra.run.smithi136.stdout: - /flannel 2022-03-02T06:39:54.307 INFO:teuthology.orchestra.run.smithi136.stdout: - /opt/cni/bin/flannel 2022-03-02T06:39:54.307 INFO:teuthology.orchestra.run.smithi136.stdout: volumeMounts: 2022-03-02T06:39:54.307 INFO:teuthology.orchestra.run.smithi136.stdout: - name: cni-plugin 2022-03-02T06:39:54.308 INFO:teuthology.orchestra.run.smithi136.stdout: mountPath: /opt/cni/bin 2022-03-02T06:39:54.308 INFO:teuthology.orchestra.run.smithi136.stdout: - name: install-cni 2022-03-02T06:39:54.309 INFO:teuthology.orchestra.run.smithi136.stdout: #image: flannelcni/flannel:v0.16.3 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-02T06:39:54.309 INFO:teuthology.orchestra.run.smithi136.stdout: image: rancher/mirrored-flannelcni-flannel:v0.16.3 2022-03-02T06:39:54.309 INFO:teuthology.orchestra.run.smithi136.stdout: command: 2022-03-02T06:39:54.309 INFO:teuthology.orchestra.run.smithi136.stdout: - cp 2022-03-02T06:39:54.309 INFO:teuthology.orchestra.run.smithi136.stdout: args: 2022-03-02T06:39:54.310 INFO:teuthology.orchestra.run.smithi136.stdout: - -f 2022-03-02T06:39:54.310 INFO:teuthology.orchestra.run.smithi136.stdout: - /etc/kube-flannel/cni-conf.json 2022-03-02T06:39:54.311 INFO:teuthology.orchestra.run.smithi136.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 120k 0 --:--:-- --:--:-- --:--:-- 120k 2022-03-02T06:39:54.312 INFO:teuthology.orchestra.run.smithi136.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-03-02T06:39:54.312 INFO:teuthology.orchestra.run.smithi136.stdout: volumeMounts: 2022-03-02T06:39:54.312 INFO:teuthology.orchestra.run.smithi136.stdout: - name: cni 2022-03-02T06:39:54.313 INFO:teuthology.orchestra.run.smithi136.stdout: mountPath: /etc/cni/net.d 2022-03-02T06:39:54.313 INFO:teuthology.orchestra.run.smithi136.stdout: - name: flannel-cfg 2022-03-02T06:39:54.313 INFO:teuthology.orchestra.run.smithi136.stdout: mountPath: /etc/kube-flannel/ 2022-03-02T06:39:54.313 INFO:teuthology.orchestra.run.smithi136.stdout: containers: 2022-03-02T06:39:54.314 INFO:teuthology.orchestra.run.smithi136.stdout: - name: kube-flannel 2022-03-02T06:39:54.314 INFO:teuthology.orchestra.run.smithi136.stdout: #image: flannelcni/flannel:v0.16.3 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-02T06:39:54.314 INFO:teuthology.orchestra.run.smithi136.stdout: image: rancher/mirrored-flannelcni-flannel:v0.16.3 2022-03-02T06:39:54.314 INFO:teuthology.orchestra.run.smithi136.stdout: command: 2022-03-02T06:39:54.315 INFO:teuthology.orchestra.run.smithi136.stdout: - /opt/bin/flanneld 2022-03-02T06:39:54.315 INFO:teuthology.orchestra.run.smithi136.stdout: args: 2022-03-02T06:39:54.315 INFO:teuthology.orchestra.run.smithi136.stdout: - --ip-masq 2022-03-02T06:39:54.315 INFO:teuthology.orchestra.run.smithi136.stdout: - --kube-subnet-mgr 2022-03-02T06:39:54.316 INFO:teuthology.orchestra.run.smithi136.stdout: resources: 2022-03-02T06:39:54.316 INFO:teuthology.orchestra.run.smithi136.stdout: requests: 2022-03-02T06:39:54.316 INFO:teuthology.orchestra.run.smithi136.stdout: cpu: "100m" 2022-03-02T06:39:54.316 INFO:teuthology.orchestra.run.smithi136.stdout: memory: "50Mi" 2022-03-02T06:39:54.317 INFO:teuthology.orchestra.run.smithi136.stdout: limits: 2022-03-02T06:39:54.317 INFO:teuthology.orchestra.run.smithi136.stdout: cpu: "100m" 2022-03-02T06:39:54.317 INFO:teuthology.orchestra.run.smithi136.stdout: memory: "50Mi" 2022-03-02T06:39:54.317 INFO:teuthology.orchestra.run.smithi136.stdout: securityContext: 2022-03-02T06:39:54.318 INFO:teuthology.orchestra.run.smithi136.stdout: privileged: false 2022-03-02T06:39:54.318 INFO:teuthology.orchestra.run.smithi136.stdout: capabilities: 2022-03-02T06:39:54.318 INFO:teuthology.orchestra.run.smithi136.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-03-02T06:39:54.318 INFO:teuthology.orchestra.run.smithi136.stdout: env: 2022-03-02T06:39:54.319 INFO:teuthology.orchestra.run.smithi136.stdout: - name: POD_NAME 2022-03-02T06:39:54.319 INFO:teuthology.orchestra.run.smithi136.stdout: valueFrom: 2022-03-02T06:39:54.319 INFO:teuthology.orchestra.run.smithi136.stdout: fieldRef: 2022-03-02T06:39:54.319 INFO:teuthology.orchestra.run.smithi136.stdout: fieldPath: metadata.name 2022-03-02T06:39:54.319 INFO:teuthology.orchestra.run.smithi136.stdout: - name: POD_NAMESPACE 2022-03-02T06:39:54.320 INFO:teuthology.orchestra.run.smithi136.stdout: valueFrom: 2022-03-02T06:39:54.320 INFO:teuthology.orchestra.run.smithi136.stdout: fieldRef: 2022-03-02T06:39:54.320 INFO:teuthology.orchestra.run.smithi136.stdout: fieldPath: metadata.namespace 2022-03-02T06:39:54.320 INFO:teuthology.orchestra.run.smithi136.stdout: volumeMounts: 2022-03-02T06:39:54.321 INFO:teuthology.orchestra.run.smithi136.stdout: - name: run 2022-03-02T06:39:54.321 INFO:teuthology.orchestra.run.smithi136.stdout: mountPath: /run/flannel 2022-03-02T06:39:54.321 INFO:teuthology.orchestra.run.smithi136.stdout: - name: flannel-cfg 2022-03-02T06:39:54.321 INFO:teuthology.orchestra.run.smithi136.stdout: mountPath: /etc/kube-flannel/ 2022-03-02T06:39:54.321 INFO:teuthology.orchestra.run.smithi136.stdout: - name: xtables-lock 2022-03-02T06:39:54.322 INFO:teuthology.orchestra.run.smithi136.stdout: mountPath: /run/xtables.lock 2022-03-02T06:39:54.322 INFO:teuthology.orchestra.run.smithi136.stdout: volumes: 2022-03-02T06:39:54.322 INFO:teuthology.orchestra.run.smithi136.stdout: - name: run 2022-03-02T06:39:54.322 INFO:teuthology.orchestra.run.smithi136.stdout: hostPath: 2022-03-02T06:39:54.323 INFO:teuthology.orchestra.run.smithi136.stdout: path: /run/flannel 2022-03-02T06:39:54.323 INFO:teuthology.orchestra.run.smithi136.stdout: - name: cni-plugin 2022-03-02T06:39:54.323 INFO:teuthology.orchestra.run.smithi136.stdout: hostPath: 2022-03-02T06:39:54.323 INFO:teuthology.orchestra.run.smithi136.stdout: path: /opt/cni/bin 2022-03-02T06:39:54.324 INFO:teuthology.orchestra.run.smithi136.stdout: - name: cni 2022-03-02T06:39:54.324 INFO:teuthology.orchestra.run.smithi136.stdout: hostPath: 2022-03-02T06:39:54.324 INFO:teuthology.orchestra.run.smithi136.stdout: path: /etc/cni/net.d 2022-03-02T06:39:54.324 INFO:teuthology.orchestra.run.smithi136.stdout: - name: flannel-cfg 2022-03-02T06:39:54.324 INFO:teuthology.orchestra.run.smithi136.stdout: configMap: 2022-03-02T06:39:54.325 INFO:teuthology.orchestra.run.smithi136.stdout: name: kube-flannel-cfg 2022-03-02T06:39:54.325 INFO:teuthology.orchestra.run.smithi136.stdout: - name: xtables-lock 2022-03-02T06:39:54.325 INFO:teuthology.orchestra.run.smithi136.stdout: hostPath: 2022-03-02T06:39:54.325 INFO:teuthology.orchestra.run.smithi136.stdout: path: /run/xtables.lock 2022-03-02T06:39:54.326 INFO:teuthology.orchestra.run.smithi136.stdout: type: FileOrCreate 2022-03-02T06:39:54.354 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-02T06:39:54.355 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.252.56.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-02T06:39:54.370 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.252.56.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.16.3 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.16.3 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-03-02T06:39:54.371 DEBUG:teuthology.orchestra.run.smithi136:> kubectl apply -f - 2022-03-02T06:39:57.023 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-02T06:39:57.028 INFO:teuthology.orchestra.run.smithi136.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-03-02T06:39:57.065 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-03-02T06:39:57.074 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-03-02T06:39:57.084 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/flannel created 2022-03-02T06:39:57.093 INFO:teuthology.orchestra.run.smithi136.stdout:configmap/kube-flannel-cfg created 2022-03-02T06:39:57.152 INFO:teuthology.orchestra.run.smithi136.stdout:daemonset.apps/kube-flannel-ds created 2022-03-02T06:39:57.157 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-02T06:39:57.157 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:39:57.158 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2022-03-02T06:39:57.164 INFO:tasks.kubeadm:Scratch devices on smithi136: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-02T06:39:57.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-02T06:39:57.187 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records in 2022-03-02T06:39:57.188 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2022-03-02T06:39:57.188 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0147981 s, 709 MB/s 2022-03-02T06:39:57.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-02T06:39:57.258 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records in 2022-03-02T06:39:57.258 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2022-03-02T06:39:57.259 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123372 s, 850 MB/s 2022-03-02T06:39:57.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-02T06:39:57.281 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records in 2022-03-02T06:39:57.282 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2022-03-02T06:39:57.282 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113876 s, 921 MB/s 2022-03-02T06:39:57.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-02T06:39:57.309 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records in 2022-03-02T06:39:57.310 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2022-03-02T06:39:57.310 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137838 s, 761 MB/s 2022-03-02T06:39:57.318 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-02T06:39:57.318 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi136-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi136 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi136-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi136 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi136-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi136 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi136-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi136 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-02T06:39:57.318 DEBUG:teuthology.orchestra.run.smithi136:> kubectl create -f - 2022-03-02T06:39:57.593 INFO:teuthology.orchestra.run.smithi136.stdout:persistentvolume/smithi136-lv-1 created 2022-03-02T06:39:57.597 INFO:teuthology.orchestra.run.smithi136.stdout:persistentvolume/smithi136-lv-2 created 2022-03-02T06:39:57.605 INFO:teuthology.orchestra.run.smithi136.stdout:persistentvolume/smithi136-lv-3 created 2022-03-02T06:39:57.613 INFO:teuthology.orchestra.run.smithi136.stdout:persistentvolume/smithi136-lv-4 created 2022-03-02T06:39:57.696 INFO:teuthology.orchestra.run.smithi136.stdout:storageclass.storage.k8s.io/scratch created 2022-03-02T06:39:57.704 DEBUG:teuthology.orchestra.run.smithi136:> kubectl taint node smithi136 node-role.kubernetes.io/master- || true 2022-03-02T06:39:57.783 INFO:teuthology.orchestra.run.smithi136.stdout:node/smithi136 untainted 2022-03-02T06:39:57.786 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-02T06:39:57.786 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-02T06:39:57.798 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-02T06:39:57.799 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:39:57.799 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2022-03-02T06:39:57.805 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-02T06:39:57.805 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-02T06:39:57.855 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-02T06:39:57.856 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-02T06:39:57.856 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-02T06:39:57.856 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-02T06:39:57.857 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2022-03-02 06:39:57.185533004 +0000 2022-03-02T06:39:57.857 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2022-03-02 06:39:57.185533004 +0000 2022-03-02T06:39:57.857 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2022-03-02 06:39:57.185533004 +0000 2022-03-02T06:39:57.857 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2022-03-02T06:39:57.865 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2022-03-02T06:39:57.865 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2022-03-02T06:39:57.866 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000660957 s, 775 kB/s 2022-03-02T06:39:57.869 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-02T06:39:57.876 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-02T06:39:57.876 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-02T06:39:57.877 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-02T06:39:57.877 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-02T06:39:57.877 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2022-03-02 06:39:57.257533003 +0000 2022-03-02T06:39:57.877 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2022-03-02 06:39:57.257533003 +0000 2022-03-02T06:39:57.877 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2022-03-02 06:39:57.257533003 +0000 2022-03-02T06:39:57.878 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2022-03-02T06:39:57.884 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2022-03-02T06:39:57.884 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2022-03-02T06:39:57.884 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000552518 s, 927 kB/s 2022-03-02T06:39:57.888 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-02T06:39:57.895 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-02T06:39:57.895 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-02T06:39:57.895 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-02T06:39:57.896 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-02T06:39:57.896 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2022-03-02 06:39:57.277533003 +0000 2022-03-02T06:39:57.896 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2022-03-02 06:39:57.277533003 +0000 2022-03-02T06:39:57.896 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2022-03-02 06:39:57.277533003 +0000 2022-03-02T06:39:57.897 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2022-03-02T06:39:57.904 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2022-03-02T06:39:57.904 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2022-03-02T06:39:57.905 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000615194 s, 832 kB/s 2022-03-02T06:39:57.909 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-02T06:39:57.960 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-02T06:39:57.960 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-02T06:39:57.961 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-03-02T06:39:57.961 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-02T06:39:57.961 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2022-03-02 06:39:57.309533003 +0000 2022-03-02T06:39:57.962 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2022-03-02 06:39:57.309533003 +0000 2022-03-02T06:39:57.962 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2022-03-02 06:39:57.309533003 +0000 2022-03-02T06:39:57.963 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2022-03-02T06:39:57.969 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2022-03-02T06:39:57.969 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2022-03-02T06:39:57.969 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000577043 s, 887 kB/s 2022-03-02T06:39:57.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-02T06:39:58.040 INFO:teuthology.orchestra.run.smithi136.stdout:loop 2022-03-02T06:39:58.042 INFO:tasks.nvme_loop:Connecting nvme_loop smithi136:/dev/vg_nvme/lv_1... 2022-03-02T06:39:58.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-02T06:39:58.063 INFO:teuthology.orchestra.run.smithi136.stdout:1 2022-03-02T06:39:58.088 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/vg_nvme/lv_11 2022-03-02T06:39:58.106 INFO:tasks.nvme_loop:Connecting nvme_loop smithi136:/dev/vg_nvme/lv_2... 2022-03-02T06:39:58.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-02T06:39:58.130 INFO:teuthology.orchestra.run.smithi136.stdout:1 2022-03-02T06:39:58.160 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/vg_nvme/lv_21 2022-03-02T06:39:58.181 INFO:tasks.nvme_loop:Connecting nvme_loop smithi136:/dev/vg_nvme/lv_3... 2022-03-02T06:39:58.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-02T06:39:58.201 INFO:teuthology.orchestra.run.smithi136.stdout:1 2022-03-02T06:39:58.226 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/vg_nvme/lv_31 2022-03-02T06:39:58.246 INFO:tasks.nvme_loop:Connecting nvme_loop smithi136:/dev/vg_nvme/lv_4... 2022-03-02T06:39:58.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-02T06:39:58.307 INFO:teuthology.orchestra.run.smithi136.stdout:1 2022-03-02T06:39:58.329 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/vg_nvme/lv_41 2022-03-02T06:39:58.348 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:39:58.348 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2022-03-02T06:39:58.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo nvme list 2022-03-02T06:39:58.366 INFO:teuthology.orchestra.run.smithi136.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-02T06:39:58.366 INFO:teuthology.orchestra.run.smithi136.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-02T06:39:58.366 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme0n1 CVPD514100122P0SGN INTEL SSDPEDMX020T4 1 2.00 TB / 2.00 TB 512 B + 0 B 8DV10171 2022-03-02T06:39:58.367 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme1n1 879175f249b70a06 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-02T06:39:58.367 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme2n1 7c81dd698ed0c4e8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-02T06:39:58.368 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme3n1 3c93b782a2deac77 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-02T06:39:58.369 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-02T06:39:59.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo nvme list 2022-03-02T06:39:59.387 INFO:teuthology.orchestra.run.smithi136.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-02T06:39:59.387 INFO:teuthology.orchestra.run.smithi136.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-02T06:39:59.388 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme0n1 CVPD514100122P0SGN INTEL SSDPEDMX020T4 1 2.00 TB / 2.00 TB 512 B + 0 B 8DV10171 2022-03-02T06:39:59.388 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme1n1 879175f249b70a06 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-02T06:39:59.388 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme2n1 7c81dd698ed0c4e8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-02T06:39:59.388 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme3n1 3c93b782a2deac77 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-02T06:39:59.389 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/nvme4n1 883b5ae28077ed76 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-02T06:39:59.390 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-02T06:39:59.390 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:39:59.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/scratch_devs 2022-03-02T06:39:59.405 INFO:teuthology.run_tasks:Running task rook... 2022-03-02T06:39:59.483 INFO:tasks.rook:Rook start 2022-03-02T06:39:59.483 INFO:tasks.rook:Config: {'conf': {'global': {'osd crush chooseleaf type': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2', 'rook_branch': 'v1.7.2', 'rook_image': 'rook/ceph:v1.7.2'} 2022-03-02T06:39:59.484 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 2022-03-02T06:39:59.484 INFO:tasks.rook:Cloning https://github.com/rook/rook branch v1.7.2 2022-03-02T06:39:59.485 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf rook && git clone --single-branch --branch v1.7.2 https://github.com/rook/rook rook 2022-03-02T06:39:59.494 INFO:teuthology.orchestra.run.smithi136.stderr:Cloning into 'rook'... 2022-03-02T06:40:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:Note: switching to '1784489964d2e417b6050c7edc8f626b5cca49f5'. 2022-03-02T06:40:01.730 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:40:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-02T06:40:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-02T06:40:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:state without impacting any branches by switching back to a branch. 2022-03-02T06:40:01.731 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:40:01.732 INFO:teuthology.orchestra.run.smithi136.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-02T06:40:01.732 INFO:teuthology.orchestra.run.smithi136.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-02T06:40:01.732 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:40:01.732 INFO:teuthology.orchestra.run.smithi136.stderr: git switch -c 2022-03-02T06:40:01.733 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:40:01.733 INFO:teuthology.orchestra.run.smithi136.stderr:Or undo this operation with: 2022-03-02T06:40:01.733 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:40:01.733 INFO:teuthology.orchestra.run.smithi136.stderr: git switch - 2022-03-02T06:40:01.734 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:40:01.734 INFO:teuthology.orchestra.run.smithi136.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-02T06:40:01.734 INFO:teuthology.orchestra.run.smithi136.stderr: 2022-03-02T06:40:01.828 INFO:tasks.rook:/home/teuthworker 2022-03-02T06:40:01.828 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/sbin:/usr/sbin ip addr show 2022-03-02T06:40:01.876 INFO:teuthology.orchestra.run.smithi136.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-02T06:40:01.876 INFO:teuthology.orchestra.run.smithi136.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-02T06:40:01.876 INFO:teuthology.orchestra.run.smithi136.stdout: inet 127.0.0.1/8 scope host lo 2022-03-02T06:40:01.877 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2022-03-02T06:40:01.877 INFO:teuthology.orchestra.run.smithi136.stdout: inet6 ::1/128 scope host 2022-03-02T06:40:01.877 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2022-03-02T06:40:01.877 INFO:teuthology.orchestra.run.smithi136.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-02T06:40:01.878 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:d9:3b:1a brd ff:ff:ff:ff:ff:ff 2022-03-02T06:40:01.878 INFO:teuthology.orchestra.run.smithi136.stdout: altname enp7s0f0 2022-03-02T06:40:01.878 INFO:teuthology.orchestra.run.smithi136.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-02T06:40:01.878 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:d9:3b:1b brd ff:ff:ff:ff:ff:ff 2022-03-02T06:40:01.878 INFO:teuthology.orchestra.run.smithi136.stdout: altname enp7s0f1 2022-03-02T06:40:01.879 INFO:teuthology.orchestra.run.smithi136.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-02T06:40:01.879 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:88:6e:42 brd ff:ff:ff:ff:ff:ff 2022-03-02T06:40:01.879 INFO:teuthology.orchestra.run.smithi136.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-02T06:40:01.879 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:88:6e:43 brd ff:ff:ff:ff:ff:ff 2022-03-02T06:40:01.880 INFO:teuthology.orchestra.run.smithi136.stdout: inet 172.21.15.136/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-02T06:40:01.880 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft 42911sec preferred_lft 42911sec 2022-03-02T06:40:01.880 INFO:teuthology.orchestra.run.smithi136.stdout: inet6 fe80::ec4:7aff:fe88:6e43/64 scope link 2022-03-02T06:40:01.880 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2022-03-02T06:40:01.880 INFO:teuthology.orchestra.run.smithi136.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-02T06:40:01.881 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 02:42:d1:8d:fb:07 brd ff:ff:ff:ff:ff:ff 2022-03-02T06:40:01.881 INFO:teuthology.orchestra.run.smithi136.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-02T06:40:01.881 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2022-03-02T06:40:01.882 DEBUG:teuthology.orchestra.run.smithi136:> which systemctl 2022-03-02T06:40:01.887 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/systemctl 2022-03-02T06:40:01.888 DEBUG:teuthology.orchestra.run.smithi136:> true 2022-03-02T06:40:01.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo lsb_release -is 2022-03-02T06:40:02.007 INFO:teuthology.orchestra.run.smithi136.stdout:Ubuntu 2022-03-02T06:40:02.013 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-02T06:40:02.013 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-02T06:40:02.033 DEBUG:teuthology.orchestra.remote:smithi136:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-02T06:40:02.047 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:40:02.047 DEBUG:teuthology.orchestra.run.smithi136:> dd if=rook/cluster/examples/kubernetes/ceph/operator.yaml of=/dev/stdout 2022-03-02T06:40:02.056 INFO:tasks.rook:Patching operator to use image rook/ceph:v1.7.2 2022-03-02T06:40:02.096 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:40:02.096 DEBUG:teuthology.orchestra.run.smithi136:> dd of=operator.yaml 2022-03-02T06:40:02.107 INFO:tasks.rook:Deploying operator 2022-03-02T06:40:02.125 DEBUG:teuthology.orchestra.remote:smithi136:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-02T06:40:02.196 DEBUG:teuthology.orchestra.remote:smithi136:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-02T06:40:02.250 DEBUG:teuthology.orchestra.run.smithi136:> kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml 2022-03-02T06:40:02.522 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-02T06:40:02.545 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-02T06:40:02.703 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-02T06:40:02.727 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-02T06:40:02.835 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-02T06:40:02.870 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-02T06:40:02.888 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-02T06:40:03.007 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-02T06:40:03.022 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-02T06:40:03.047 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-02T06:40:03.073 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-02T06:40:03.177 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-02T06:40:03.205 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-02T06:40:03.250 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-02T06:40:03.373 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created 2022-03-02T06:40:03.408 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created 2022-03-02T06:40:03.432 INFO:teuthology.orchestra.run.smithi136.stdout:customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created 2022-03-02T06:40:03.522 INFO:teuthology.orchestra.run.smithi136.stdout:namespace/rook-ceph created 2022-03-02T06:40:03.527 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-02T06:40:03.555 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-ceph-admission-controller created 2022-03-02T06:40:03.564 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role created 2022-03-02T06:40:03.569 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding created 2022-03-02T06:40:03.597 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-02T06:40:03.678 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-02T06:40:03.682 INFO:teuthology.orchestra.run.smithi136.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-02T06:40:03.695 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-02T06:40:03.723 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-02T06:40:03.747 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-02T06:40:03.752 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-ceph-system created 2022-03-02T06:40:03.761 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-02T06:40:03.811 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-02T06:40:03.822 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-02T06:40:03.827 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-ceph-osd created 2022-03-02T06:40:03.836 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-ceph-mgr created 2022-03-02T06:40:03.864 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-02T06:40:03.870 INFO:teuthology.orchestra.run.smithi136.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-02T06:40:03.898 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-02T06:40:03.981 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-02T06:40:04.416 INFO:teuthology.orchestra.run.smithi136.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-02T06:40:04.417 INFO:teuthology.orchestra.run.smithi136.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-02T06:40:04.417 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-02T06:40:04.417 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-02T06:40:04.417 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-02T06:40:04.418 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-02T06:40:04.418 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-02T06:40:04.418 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-02T06:40:04.418 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-02T06:40:04.419 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-02T06:40:04.421 INFO:teuthology.orchestra.run.smithi136.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-02T06:40:04.421 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-02T06:40:04.421 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-02T06:40:04.422 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-02T06:40:04.422 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-02T06:40:04.422 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-02T06:40:04.422 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-02T06:40:04.422 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-02T06:40:04.423 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-02T06:40:04.423 INFO:teuthology.orchestra.run.smithi136.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-02T06:40:04.423 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-02T06:40:04.423 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-02T06:40:04.424 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-02T06:40:04.424 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-02T06:40:04.424 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-02T06:40:04.458 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-02T06:40:04.460 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-02T06:40:04.470 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-02T06:40:04.490 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-02T06:40:04.503 INFO:teuthology.orchestra.run.smithi136.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-02T06:40:04.511 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-02T06:40:04.520 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-02T06:40:04.528 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-02T06:40:04.549 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-02T06:40:04.612 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-02T06:40:04.616 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-02T06:40:04.628 INFO:teuthology.orchestra.run.smithi136.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-02T06:40:04.636 INFO:teuthology.orchestra.run.smithi136.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-02T06:40:04.644 INFO:teuthology.orchestra.run.smithi136.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-02T06:40:04.665 INFO:teuthology.orchestra.run.smithi136.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-02T06:40:04.678 INFO:teuthology.orchestra.run.smithi136.stdout:configmap/rook-ceph-operator-config created 2022-03-02T06:40:04.700 INFO:teuthology.orchestra.run.smithi136.stdout:deployment.apps/rook-ceph-operator created 2022-03-02T06:40:04.705 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-02T06:40:04.787 INFO:teuthology.orchestra.run.smithi136.stderr:No resources found in rook-ceph namespace. 2022-03-02T06:40:14.791 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-02T06:40:14.856 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:40:14.856 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-operator-6b496c8959-ndxt2 0/1 Pending 0 8s 2022-03-02T06:40:24.860 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-02T06:40:24.923 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:40:24.923 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-operator-6b496c8959-ndxt2 0/1 Pending 0 18s 2022-03-02T06:40:34.927 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-02T06:40:34.989 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:40:34.989 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-operator-6b496c8959-ndxt2 0/1 ContainerCreating 0 28s 2022-03-02T06:40:44.992 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-02T06:40:45.049 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:40:45.049 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-operator-6b496c8959-ndxt2 0/1 ContainerCreating 0 39s 2022-03-02T06:40:55.053 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-02T06:40:55.487 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:40:55.488 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-operator-6b496c8959-ndxt2 0/1 ContainerCreating 0 49s 2022-03-02T06:41:05.491 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-02T06:41:05.561 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:05.562 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-operator-6b496c8959-ndxt2 1/1 Running 0 59s 2022-03-02T06:41:05.563 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph logs -f rook-ceph-operator-6b496c8959-ndxt2 2022-03-02T06:41:05.573 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:41:05.573 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2022-03-02T06:41:05.583 INFO:tasks.rook: override: [global] osd crush chooseleaf type = 0 2022-03-02T06:41:05.584 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-02T06:41:05.584 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-02T06:41:05.584 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-02T06:41:05.585 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-02T06:41:05.585 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-02T06:41:05.585 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-02T06:41:05.586 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-02T06:41:05.586 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-02T06:41:05.589 DEBUG:teuthology.orchestra.run.smithi136:> kubectl create -f - 2022-03-02T06:41:05.662 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.748663 I | rookcmd: starting Rook v1.7.2 with arguments '/usr/local/bin/rook ceph operator' 2022-03-02T06:41:05.662 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.748746 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule= 2022-03-02T06:41:05.663 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.748750 I | cephcmd: starting Rook-Ceph operator 2022-03-02T06:41:05.663 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.888396 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)" 2022-03-02T06:41:05.663 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.934333 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-02T06:41:05.663 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.968055 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-02T06:41:05.663 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.972794 I | operator: looking for secret "rook-ceph-admission-controller" 2022-03-02T06:41:05.664 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.975221 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-02T06:41:05.664 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.977702 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-02T06:41:05.664 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.977719 I | operator: watching all namespaces for ceph cluster CRs 2022-03-02T06:41:05.664 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.977809 I | operator: setting up the controller-runtime manager 2022-03-02T06:41:05.665 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:58.979559 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations 2022-03-02T06:41:05.665 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.005353 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-02T06:41:05.665 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.285570 I | ceph-cluster-controller: successfully started 2022-03-02T06:41:05.665 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.285692 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-02T06:41:05.666 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.285725 I | ceph-crashcollector-controller: successfully started 2022-03-02T06:41:05.666 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.285852 I | ceph-block-pool-controller: successfully started 2022-03-02T06:41:05.666 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.285952 I | ceph-object-store-user-controller: successfully started 2022-03-02T06:41:05.666 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.286051 I | ceph-object-realm-controller: successfully started 2022-03-02T06:41:05.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.286133 I | ceph-object-zonegroup-controller: successfully started 2022-03-02T06:41:05.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.286240 I | ceph-object-zone-controller: successfully started 2022-03-02T06:41:05.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.286499 I | ceph-object-controller: successfully started 2022-03-02T06:41:05.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.286647 I | ceph-file-controller: successfully started 2022-03-02T06:41:05.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.286757 I | ceph-nfs-controller: successfully started 2022-03-02T06:41:05.668 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.286890 I | ceph-rbd-mirror-controller: successfully started 2022-03-02T06:41:05.668 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.287006 I | ceph-client-controller: successfully started 2022-03-02T06:41:05.668 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.287149 I | ceph-filesystem-mirror-controller: successfully started 2022-03-02T06:41:05.669 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:40:59.288550 I | operator: starting the controller-runtime manager 2022-03-02T06:41:06.465 INFO:teuthology.orchestra.run.smithi136.stdout:configmap/rook-config-override created 2022-03-02T06:41:06.470 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 1 skipUpgradeChecks: true 2022-03-02T06:41:06.470 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:41:06.471 DEBUG:teuthology.orchestra.run.smithi136:> dd of=cluster.yaml 2022-03-02T06:41:06.476 DEBUG:teuthology.orchestra.run.smithi136:> kubectl create -f cluster.yaml 2022-03-02T06:41:08.738 INFO:teuthology.orchestra.run.smithi136.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-02T06:41:08.748 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.744487 I | clusterdisruption-controller: create event from ceph cluster CR 2022-03-02T06:41:08.748 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.744615 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-02T06:41:08.760 DEBUG:teuthology.orchestra.remote:smithi136:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-02T06:41:08.772 DEBUG:teuthology.orchestra.run.smithi136:> kubectl create -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-02T06:41:08.783 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.779736 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-02T06:41:08.783 INFO:tasks.rook.operator.smithi136.stdout: ... // 3 identical fields 2022-03-02T06:41:08.783 INFO:tasks.rook.operator.smithi136.stdout: Labels: nil, 2022-03-02T06:41:08.784 INFO:tasks.rook.operator.smithi136.stdout: Placement: nil, 2022-03-02T06:41:08.784 INFO:tasks.rook.operator.smithi136.stdout: Network: v1.NetworkSpec{ 2022-03-02T06:41:08.784 INFO:tasks.rook.operator.smithi136.stdout: Provider: "", 2022-03-02T06:41:08.784 INFO:tasks.rook.operator.smithi136.stdout: Selectors: nil, 2022-03-02T06:41:08.785 INFO:tasks.rook.operator.smithi136.stdout: HostNetwork: false, 2022-03-02T06:41:08.785 INFO:tasks.rook.operator.smithi136.stdout:- IPFamily: "", 2022-03-02T06:41:08.785 INFO:tasks.rook.operator.smithi136.stdout:+ IPFamily: "IPv4", 2022-03-02T06:41:08.785 INFO:tasks.rook.operator.smithi136.stdout: DualStack: false, 2022-03-02T06:41:08.786 INFO:tasks.rook.operator.smithi136.stdout: }, 2022-03-02T06:41:08.786 INFO:tasks.rook.operator.smithi136.stdout: Resources: nil, 2022-03-02T06:41:08.786 INFO:tasks.rook.operator.smithi136.stdout: PriorityClassNames: nil, 2022-03-02T06:41:08.786 INFO:tasks.rook.operator.smithi136.stdout: ... // 16 identical fields 2022-03-02T06:41:08.787 INFO:tasks.rook.operator.smithi136.stdout: } 2022-03-02T06:41:08.787 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.780556 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:41:08.787 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.783366 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-02T06:41:08.798 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.796367 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-02T06:41:08.827 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.824923 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster 2022-03-02T06:41:08.827 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.824950 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-02T06:41:08.830 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.827499 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-02T06:41:08.833 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.829333 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-02T06:41:08.833 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.831230 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-02T06:41:08.836 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.833113 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-02T06:41:08.839 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.835109 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.4.0" (default) 2022-03-02T06:41:08.843 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.840241 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2... 2022-03-02T06:41:08.991 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:08.986812 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0" (default) 2022-03-02T06:41:09.006 INFO:teuthology.orchestra.run.smithi136.stdout:deployment.apps/rook-ceph-tools created 2022-03-02T06:41:09.011 INFO:tasks.rook:Waiting for tools container to start 2022-03-02T06:41:09.011 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:09.156 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:09.156 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 Pending 0 0s 2022-03-02T06:41:09.385 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:09.383673 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2" (default) 2022-03-02T06:41:09.787 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:09.785522 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.2.1" (default) 2022-03-02T06:41:10.195 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:10.193219 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1" (default) 2022-03-02T06:41:10.197 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:10.196031 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-02T06:41:10.386 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:10.384050 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default) 2022-03-02T06:41:10.625 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:10.623541 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-02T06:41:10.787 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:10.785030 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-02T06:41:10.787 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:10.785066 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.4.0" 2022-03-02T06:41:10.987 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:10.985597 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-02T06:41:14.159 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:14.221 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:14.221 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 5s 2022-03-02T06:41:19.225 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:19.293 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:19.293 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 10s 2022-03-02T06:41:24.295 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:24.361 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:24.361 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 15s 2022-03-02T06:41:29.365 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:29.442 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:29.442 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 20s 2022-03-02T06:41:34.445 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:34.503 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:34.504 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 25s 2022-03-02T06:41:39.507 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:39.579 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:39.580 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 30s 2022-03-02T06:41:44.582 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:44.644 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:44.645 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 35s 2022-03-02T06:41:49.227 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.221930 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10916 quincy" 2022-03-02T06:41:49.228 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.221952 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-02T06:41:49.228 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.221959 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10916 quincy", pursuing 2022-03-02T06:41:49.477 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.475102 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10916 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2" 2022-03-02T06:41:49.479 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.476479 I | ceph-csi: Detected ceph CSI image version: "v3.4.0" 2022-03-02T06:41:49.536 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.534664 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-02T06:41:49.647 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:49.682 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.680535 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-02T06:41:49.684 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.681941 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-02 06:41:49.655939802 +0000 UTC m=+50.965740391 LastTransitionTime:2022-03-02 06:41:49.655939615 +0000 UTC m=+50.965740233}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T06:41:49.793 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:49.794 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 40s 2022-03-02T06:41:49.794 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.788261 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-02T06:41:49.795 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.791833 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-02T06:41:49.802 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.801453 I | op-mon: start running mons 2022-03-02T06:41:49.827 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.822049 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-02T06:41:49.856 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.855349 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-02T06:41:49.860 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.857842 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-02T06:41:49.863 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.860718 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-02T06:41:49.883 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:49.882700 I | op-mon: creating mon secrets for a new cluster 2022-03-02T06:41:50.009 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:50.003903 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-02T06:41:50.405 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:50.404248 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-02T06:41:50.804 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:50.803942 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-02T06:41:51.004 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:51.003719 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-02T06:41:51.206 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:51.204791 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-02T06:41:51.410 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:51.409563 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-02T06:41:51.606 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:51.605495 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-02T06:41:51.607 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:51.605515 I | ceph-csi: Kubernetes version is 1.23 2022-03-02T06:41:52.005 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:52.004460 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0" (default) 2022-03-02T06:41:52.406 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:52.405365 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-02T06:41:53.016 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:53.015391 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:41:53.017 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:53.015673 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:41:53.206 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:53.205209 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-02T06:41:53.605 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:53.604794 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-02T06:41:54.008 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:54.004639 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-02T06:41:54.407 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:54.405689 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-02T06:41:54.797 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:41:54.806 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:54.805296 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-02T06:41:54.860 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:41:54.860 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 45s 2022-03-02T06:41:55.206 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:55.204666 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-02T06:41:55.409 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:55.407712 I | op-mon: targeting the mon count 1 2022-03-02T06:41:55.423 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:55.421605 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-02T06:41:55.423 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:55.421691 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:41:55.606 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:55.604283 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-02T06:41:56.020 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:56.018184 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-02T06:41:56.206 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:56.204220 I | op-mon: parsing mon endpoints: 2022-03-02T06:41:56.207 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:56.204250 W | op-mon: ignoring invalid monitor 2022-03-02T06:41:56.218 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:56.215857 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2... 2022-03-02T06:41:56.572 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:56.570736 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-02T06:41:57.003 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:57.000317 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-02T06:41:57.244 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:57.242234 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-02T06:41:57.563 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:57.560507 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-02T06:41:58.122 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:58.115826 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-02T06:41:58.215 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:58.211246 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-02T06:41:58.534 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:58.530923 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-02T06:41:58.702 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:58.698743 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-02T06:41:58.822 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:58.704586 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-02T06:41:59.138 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:59.133759 I | ceph-csi: successfully started CSI CephFS driver 2022-03-02T06:41:59.645 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:41:59.639870 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-02T06:41:59.863 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:42:00.273 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:42:00.273 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 51s 2022-03-02T06:42:00.274 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:00.269964 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-02T06:42:00.539 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:00.535300 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-03-02T06:42:00.601 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:00.597396 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-02T06:42:05.276 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:42:05.543 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:42:05.544 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 56s 2022-03-02T06:42:10.546 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:42:10.696 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:42:10.697 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 61s 2022-03-02T06:42:11.644 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:11.638867 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10916 quincy" 2022-03-02T06:42:11.644 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:11.638896 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-02T06:42:11.645 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:11.638907 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10916 quincy", pursuing 2022-03-02T06:42:11.675 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:11.670183 I | op-mon: parsing mon endpoints: 2022-03-02T06:42:11.675 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:11.670216 W | op-mon: ignoring invalid monitor 2022-03-02T06:42:12.064 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:12.057377 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:42:12.066 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:12.058666 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:42:12.176 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:12.171279 E | ceph-cluster-controller: failed to get ceph daemons versions, this typically happens during the first cluster initialization. failed to run 'ceph versions'. . unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-02T06:42:12.176 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T06:42:12.165+0000 7f6ddb4a4700 -1 failed for service _ceph-mon._tcp 2022-03-02T06:42:12.176 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T06:42:12.165+0000 7f6ddb4a4700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-02T06:42:12.176 INFO:tasks.rook.operator.smithi136.stdout:[errno 2] RADOS object not found (error connecting to the cluster): exit status 1 2022-03-02T06:42:12.177 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:12.171303 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10916 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2" 2022-03-02T06:42:13.011 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:13.006680 I | op-mon: start running mons 2022-03-02T06:42:13.062 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:13.058353 I | op-mon: parsing mon endpoints: 2022-03-02T06:42:13.063 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:13.058371 W | op-mon: ignoring invalid monitor 2022-03-02T06:42:13.427 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:13.422552 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-02T06:42:13.467 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:13.463461 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:42:13.528 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:13.524205 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:42:14.142 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:14.138391 I | op-mon: targeting the mon count 1 2022-03-02T06:42:14.149 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:14.145138 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-02T06:42:14.413 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:14.408654 I | op-mon: sched-mon: waiting for canary pod creation rook-ceph-mon-a-canary 2022-03-02T06:42:15.699 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:42:15.752 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:42:15.753 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 66s 2022-03-02T06:42:19.644 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:19.639711 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi136 2022-03-02T06:42:19.644 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:19.639741 I | op-mon: assignmon: mon a assigned to node smithi136 2022-03-02T06:42:19.648 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:19.643846 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-02T06:42:19.651 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:19.647669 I | op-mon: creating mon a 2022-03-02T06:42:20.755 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:42:20.821 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:20.815734 I | op-mon: mon "a" endpoint is [v2:10.103.147.119:3300,v1:10.103.147.119:6789] 2022-03-02T06:42:20.843 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:42:20.844 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 0/1 ContainerCreating 0 71s 2022-03-02T06:42:21.124 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:21.117136 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:-1] 2022-03-02T06:42:21.124 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:21.117171 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-02T06:42:21.124 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:21.117419 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-02T06:42:21.968 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:21.854939 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:42:21.971 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:21.964523 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:42:22.267 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:22.262740 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-02T06:42:22.365 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:22.361312 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-02T06:42:23.423 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:23.419051 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:42:23.423 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:23.419088 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:42:23.453 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:23.449024 I | op-mon: mon a is not yet running 2022-03-02T06:42:23.454 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:23.449066 I | op-mon: mons running: [] 2022-03-02T06:42:25.846 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-02T06:42:26.024 INFO:teuthology.orchestra.run.smithi136.stdout:NAME READY STATUS RESTARTS AGE 2022-03-02T06:42:26.025 INFO:teuthology.orchestra.run.smithi136.stdout:rook-ceph-tools-7677f7f4b9-hdpwp 1/1 Running 0 77s 2022-03-02T06:42:26.026 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-02T06:42:26.026 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch status -f json 2022-03-02T06:42:26.665 INFO:teuthology.orchestra.run.smithi136.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-02T06:42:26.665 INFO:teuthology.orchestra.run.smithi136.stderr:2022-03-02T06:42:26.653+0000 7f699944b700 -1 failed for service _ceph-mon._tcp 2022-03-02T06:42:26.665 INFO:teuthology.orchestra.run.smithi136.stderr:2022-03-02T06:42:26.653+0000 7f699944b700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-02T06:42:26.666 INFO:teuthology.orchestra.run.smithi136.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-02T06:42:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:command terminated with exit code 1 2022-03-02T06:42:26.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-02T06:42:36.672 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch status -f json 2022-03-02T06:42:40.278 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:42:40.278 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2022-03-02T06:42:40.279 INFO:teuthology.orchestra.run.smithi136.stderr:pg stat 2022-03-02T06:42:40.279 INFO:teuthology.orchestra.run.smithi136.stderr:pg getmap 2022-03-02T06:42:40.279 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-02T06:42:40.280 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-02T06:42:40.280 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_pools_json 2022-03-02T06:42:40.280 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-pool [...] 2022-03-02T06:42:40.280 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-primary [] [...] 2022-03-02T06:42:40.281 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-osd [] [...] 2022-03-02T06:42:40.281 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls [] [...] 2022-03-02T06:42:40.281 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-02T06:42:40.281 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2022-03-02T06:42:40.282 INFO:teuthology.orchestra.run.smithi136.stderr:command terminated with exit code 22 2022-03-02T06:42:40.284 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-02T06:42:43.564 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:43.560899 I | op-mon: mons running: [a] 2022-03-02T06:42:43.779 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:43.775602 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:42:43.780 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:43.775634 I | op-mon: mons created: 1 2022-03-02T06:42:44.016 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:44.012815 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:42:44.023 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:44.019873 I | op-mon: mons running: [a] 2022-03-02T06:42:44.236 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:44.232373 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:42:44.236 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:44.232413 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:42:44.554 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:44.550628 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:42:44.555 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:44.550663 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:42:44.870 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:44.865948 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:42:44.870 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:44.865988 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:42:45.142 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:45.138188 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:42:45.142 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:45.138213 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:42:45.440 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:45.436264 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:42:45.440 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:45.436298 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:42:45.728 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:45.724274 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:42:45.728 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:45.724301 I | op-config: deleting "log file" option from the mon configuration database 2022-03-02T06:42:45.950 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:45.946942 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-02T06:42:45.951 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:45.946984 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-02T06:42:46.283 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:46.279679 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-02T06:42:46.574 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:46.571051 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-02T06:42:46.882 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:46.878600 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-02T06:42:47.310 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:47.306428 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-02T06:42:47.310 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:47.306449 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-02T06:42:47.598 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:47.594659 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-02T06:42:47.814 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:47.810906 I | cephclient: successfully enabled msgr2 protocol 2022-03-02T06:42:47.815 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:47.810949 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-02T06:42:47.815 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:47.810965 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-02T06:42:48.101 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:48.097749 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-02T06:42:48.142 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:48.138162 I | op-mgr: start running mgr 2022-03-02T06:42:48.142 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:42:48.138201 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-02T06:42:50.285 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch status -f json 2022-03-02T06:42:50.721 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:42:50.722 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2022-03-02T06:42:50.722 INFO:teuthology.orchestra.run.smithi136.stderr:pg stat 2022-03-02T06:42:50.723 INFO:teuthology.orchestra.run.smithi136.stderr:pg getmap 2022-03-02T06:42:50.723 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-02T06:42:50.723 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-02T06:42:50.723 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_pools_json 2022-03-02T06:42:50.724 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-pool [...] 2022-03-02T06:42:50.724 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-primary [] [...] 2022-03-02T06:42:50.724 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-osd [] [...] 2022-03-02T06:42:50.725 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls [] [...] 2022-03-02T06:42:50.725 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-02T06:42:50.725 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2022-03-02T06:42:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-02T06:42:50.727 INFO:teuthology.orchestra.run.smithi136.stderr:command terminated with exit code 22 2022-03-02T06:43:00.728 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch status -f json 2022-03-02T06:43:00.822 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:00.818773 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-02T06:43:00.825 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:00.821480 I | op-mgr: setting services to point to mgr "a" 2022-03-02T06:43:00.890 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:00.886392 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-02T06:43:00.891 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:00.886433 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-02T06:43:00.891 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:00.886445 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-02T06:43:00.915 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:00.911404 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-02T06:43:00.959 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:00.955714 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-02T06:43:00.960 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:00.956180 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-02T06:43:00.960 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:00.956286 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-02T06:43:01.322 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.318942 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-02T06:43:01.322 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.318969 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:43:01.335 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.331082 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-02T06:43:01.336 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.331108 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-02T06:43:01.336 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.331125 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-02T06:43:01.336 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:43:01.337 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.334544 W | cephclient: failed to enable mgr module "rook". trying again... 2022-03-02T06:43:01.338 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2022-03-02T06:43:01.338 INFO:teuthology.orchestra.run.smithi136.stderr:pg stat 2022-03-02T06:43:01.338 INFO:teuthology.orchestra.run.smithi136.stderr:pg getmap 2022-03-02T06:43:01.339 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-02T06:43:01.339 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-02T06:43:01.339 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_pools_json 2022-03-02T06:43:01.340 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-pool [...] 2022-03-02T06:43:01.340 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-primary [] [...] 2022-03-02T06:43:01.340 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-osd [] [...] 2022-03-02T06:43:01.340 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls [] [...] 2022-03-02T06:43:01.341 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-02T06:43:01.341 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2022-03-02T06:43:01.343 INFO:teuthology.orchestra.run.smithi136.stderr:command terminated with exit code 22 2022-03-02T06:43:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-02T06:43:01.355 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.352632 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-02T06:43:01.356 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.352658 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-02T06:43:01.359 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.353517 I | op-mgr: successful modules: prometheus 2022-03-02T06:43:01.359 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.354217 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Normal:ReconcileSucceeded:successfully configured CephCluster "rook-ceph/rook-ceph" 2022-03-02T06:43:01.360 INFO:tasks.rook.operator.smithi136.stdout:I0302 06:43:01.354366 7 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-02T06:43:01.360 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.354390 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:43:01.430 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.422799 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:43:01.434 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.431596 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2... 2022-03-02T06:43:01.720 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:01.716932 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-02T06:43:02.523 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:02.520034 I | op-mgr: successful modules: dashboard 2022-03-02T06:43:07.010 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:07.007235 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-02T06:43:07.155 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:07.152053 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-02T06:43:07.498 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:07.494740 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10916 quincy" 2022-03-02T06:43:07.498 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:07.494762 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-02T06:43:07.499 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:07.494770 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10916 quincy", pursuing 2022-03-02T06:43:07.952 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:07.949556 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:43:08.045 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:08.042598 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:43:08.049 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:08.043032 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:43:08.333 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:08.330716 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10916 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2" 2022-03-02T06:43:08.496 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:08.493331 I | op-mon: start running mons 2022-03-02T06:43:08.501 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:08.498235 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:43:08.518 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:08.515648 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:43:08.570 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:08.567151 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:43:08.570 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:08.567389 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:43:08.705 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:08.702268 I | cephclient: command failed for set rook backend. trying again... 2022-03-02T06:43:09.397 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:09.393998 I | op-mon: targeting the mon count 1 2022-03-02T06:43:09.401 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:09.397652 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:43:09.699 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:09.695549 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:43:09.699 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:09.695591 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:43:09.965 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:09.962224 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:43:09.965 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:09.962256 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:43:10.200 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:10.195312 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:43:10.200 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:10.195336 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:43:10.435 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:10.432722 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:43:10.436 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:10.432749 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:43:10.648 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:10.645159 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:43:10.649 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:10.645190 I | op-config: deleting "log file" option from the mon configuration database 2022-03-02T06:43:10.871 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:10.868330 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-02T06:43:10.872 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:10.868361 I | op-mon: checking for basic quorum with existing mons 2022-03-02T06:43:11.142 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:11.139461 I | op-mon: mon "a" endpoint is [v2:10.103.147.119:3300,v1:10.103.147.119:6789] 2022-03-02T06:43:11.231 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:11.228596 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:43:11.346 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch status -f json 2022-03-02T06:43:11.395 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:11.392327 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:43:11.395 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:11.392500 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:43:11.799 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:43:11.800 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2022-03-02T06:43:11.800 INFO:teuthology.orchestra.run.smithi136.stderr:pg stat 2022-03-02T06:43:11.801 INFO:teuthology.orchestra.run.smithi136.stderr:pg getmap 2022-03-02T06:43:11.801 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-02T06:43:11.802 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-02T06:43:11.802 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_pools_json 2022-03-02T06:43:11.802 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-pool [...] 2022-03-02T06:43:11.802 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-primary [] [...] 2022-03-02T06:43:11.803 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls-by-osd [] [...] 2022-03-02T06:43:11.803 INFO:teuthology.orchestra.run.smithi136.stderr:pg ls [] [...] 2022-03-02T06:43:11.803 INFO:teuthology.orchestra.run.smithi136.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-02T06:43:11.804 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2022-03-02T06:43:11.804 INFO:teuthology.orchestra.run.smithi136.stderr:command terminated with exit code 22 2022-03-02T06:43:11.806 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:11.803475 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-02T06:43:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-02T06:43:11.812 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:11.810175 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-02T06:43:11.813 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:11.810198 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:43:12.006 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:12.001762 I | op-mon: mons running: [a] 2022-03-02T06:43:12.222 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:12.219319 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:43:12.222 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:12.219338 I | op-mon: mons created: 1 2022-03-02T06:43:12.300 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:12.297051 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-02T06:43:12.444 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:12.441440 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:43:12.450 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:12.448096 I | op-mon: mons running: [a] 2022-03-02T06:43:12.673 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:12.670584 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:43:12.674 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:12.670625 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-02T06:43:12.904 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:12.900658 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-02T06:43:13.139 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:13.136361 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-02T06:43:13.369 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:13.366360 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-02T06:43:13.794 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:13.791789 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-02T06:43:13.795 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:13.791810 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-02T06:43:14.002 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:13.999444 I | cephclient: command failed for set rook backend. trying again... 2022-03-02T06:43:14.195 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:14.192624 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-02T06:43:14.441 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:14.434138 I | cephclient: successfully enabled msgr2 protocol 2022-03-02T06:43:14.441 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:14.434161 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-02T06:43:14.442 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:14.434165 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-02T06:43:14.658 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:14.655794 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-02T06:43:14.715 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:14.712469 I | op-mgr: start running mgr 2022-03-02T06:43:14.716 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:14.712495 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-02T06:43:15.225 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:15.222556 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-02T06:43:15.233 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:15.230624 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-02T06:43:15.235 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:15.232877 I | op-mgr: setting services to point to mgr "a" 2022-03-02T06:43:16.011 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.005817 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-02T06:43:16.011 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.005875 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-02T06:43:16.012 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.005888 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-02T06:43:16.061 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.058289 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-02T06:43:16.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.191177 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-02T06:43:16.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.191203 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-02T06:43:16.195 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.191209 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-02T06:43:16.660 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.658026 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-02T06:43:16.670 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.666532 I | op-mgr: successful modules: prometheus 2022-03-02T06:43:16.671 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.667612 I | op-mgr: successful modules: dashboard 2022-03-02T06:43:16.830 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.827555 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-02T06:43:16.830 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.827589 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:43:16.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:16.945569 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-02T06:43:17.580 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:17.577632 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-02T06:43:17.961 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:17.958203 I | cephclient: command failed for set rook backend. trying again... 2022-03-02T06:43:19.309 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:19.306584 I | cephclient: command failed for set rook backend. trying again... 2022-03-02T06:43:21.809 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch status -f json 2022-03-02T06:43:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:43:22.349 INFO:teuthology.orchestra.run.smithi136.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-02T06:43:22.355 INFO:teuthology.orchestra.run.smithi136.stderr:command terminated with exit code 2 2022-03-02T06:43:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-02T06:43:22.365 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:22.362683 I | op-mgr: successful modules: balancer 2022-03-02T06:43:22.918 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:22.916069 I | op-mgr: successful modules: balancer 2022-03-02T06:43:23.367 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:23.364379 I | cephclient: action set rook backend succeeded on attempt 1 2022-03-02T06:43:23.367 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:23.364398 I | op-mgr: successful modules: orchestrator modules 2022-03-02T06:43:24.719 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:24.716353 I | cephclient: action set rook backend succeeded on attempt 3 2022-03-02T06:43:24.719 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:24.716383 I | op-mgr: successful modules: orchestrator modules 2022-03-02T06:43:32.358 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch status -f json 2022-03-02T06:43:32.948 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:43:32.949 INFO:teuthology.orchestra.run.smithi136.stdout:{"available": true, "backend": "rook"} 2022-03-02T06:43:32.990 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-02T06:43:32.991 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-02T06:43:33.572 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch apply osd --all-available-devices 2022-03-02T06:43:34.234 INFO:teuthology.orchestra.run.smithi136.stdout:['Success'] 2022-03-02T06:43:34.238 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:34.232822 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-02T06:43:34.238 INFO:tasks.rook.operator.smithi136.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", AllowUnsupported: true}, 2022-03-02T06:43:34.239 INFO:tasks.rook.operator.smithi136.stdout: Storage: v1.StorageScopeSpec{ 2022-03-02T06:43:34.239 INFO:tasks.rook.operator.smithi136.stdout: ... // 3 identical fields 2022-03-02T06:43:34.239 INFO:tasks.rook.operator.smithi136.stdout: Config: nil, 2022-03-02T06:43:34.239 INFO:tasks.rook.operator.smithi136.stdout: Selection: {}, 2022-03-02T06:43:34.240 INFO:tasks.rook.operator.smithi136.stdout:- StorageClassDeviceSets: nil, 2022-03-02T06:43:34.240 INFO:tasks.rook.operator.smithi136.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-02T06:43:34.241 INFO:tasks.rook.operator.smithi136.stdout:+ { 2022-03-02T06:43:34.241 INFO:tasks.rook.operator.smithi136.stdout:+ Name: "smithi136-lv-1", 2022-03-02T06:43:34.242 INFO:tasks.rook.operator.smithi136.stdout:+ Count: 1, 2022-03-02T06:43:34.242 INFO:tasks.rook.operator.smithi136.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-02T06:43:34.242 INFO:tasks.rook.operator.smithi136.stdout:+ }, 2022-03-02T06:43:34.242 INFO:tasks.rook.operator.smithi136.stdout:+ { 2022-03-02T06:43:34.243 INFO:tasks.rook.operator.smithi136.stdout:+ Name: "smithi136-lv-2", 2022-03-02T06:43:34.243 INFO:tasks.rook.operator.smithi136.stdout:+ Count: 1, 2022-03-02T06:43:34.243 INFO:tasks.rook.operator.smithi136.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-02T06:43:34.244 INFO:tasks.rook.operator.smithi136.stdout:+ }, 2022-03-02T06:43:34.244 INFO:tasks.rook.operator.smithi136.stdout:+ { 2022-03-02T06:43:34.244 INFO:tasks.rook.operator.smithi136.stdout:+ Name: "smithi136-lv-3", 2022-03-02T06:43:34.245 INFO:tasks.rook.operator.smithi136.stdout:+ Count: 1, 2022-03-02T06:43:34.245 INFO:tasks.rook.operator.smithi136.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-02T06:43:34.246 INFO:tasks.rook.operator.smithi136.stdout:+ }, 2022-03-02T06:43:34.246 INFO:tasks.rook.operator.smithi136.stdout:+ { 2022-03-02T06:43:34.246 INFO:tasks.rook.operator.smithi136.stdout:+ Name: "smithi136-lv-4", 2022-03-02T06:43:34.247 INFO:tasks.rook.operator.smithi136.stdout:+ Count: 1, 2022-03-02T06:43:34.247 INFO:tasks.rook.operator.smithi136.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-02T06:43:34.248 INFO:tasks.rook.operator.smithi136.stdout:+ }, 2022-03-02T06:43:34.248 INFO:tasks.rook.operator.smithi136.stdout:+ }, 2022-03-02T06:43:34.249 INFO:tasks.rook.operator.smithi136.stdout: }, 2022-03-02T06:43:34.249 INFO:tasks.rook.operator.smithi136.stdout: Annotations: nil, 2022-03-02T06:43:34.249 INFO:tasks.rook.operator.smithi136.stdout: Labels: nil, 2022-03-02T06:43:34.250 INFO:tasks.rook.operator.smithi136.stdout: ... // 20 identical fields 2022-03-02T06:43:34.252 INFO:tasks.rook.operator.smithi136.stdout: } 2022-03-02T06:43:34.254 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:34.232948 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:43:34.255 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:34.247073 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:43:34.257 INFO:tasks.rook:Waiting for 4 OSDs 2022-03-02T06:43:34.258 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:43:34.264 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:34.260226 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2... 2022-03-02T06:43:34.777 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:43:34.795 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-02T06:43:34.795 INFO:tasks.rook: have 0/4 OSDs 2022-03-02T06:43:40.420 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:40.414200 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10916 quincy" 2022-03-02T06:43:40.420 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:40.414253 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-02T06:43:40.420 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:40.414265 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10916 quincy", pursuing 2022-03-02T06:43:40.641 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:40.639098 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:43:40.802 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:40.797755 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:43:40.805 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:40.800754 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:43:41.215 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:41.213238 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10916 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2" 2022-03-02T06:43:41.372 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:41.369265 I | op-mon: start running mons 2022-03-02T06:43:41.563 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:41.561838 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:43:41.789 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:41.787264 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:43:41.795 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:41.793855 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:43:41.796 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:41.794018 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:43:42.281 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:42.279195 I | op-mon: targeting the mon count 1 2022-03-02T06:43:42.297 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:42.295657 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-02T06:43:43.298 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:43.296184 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:43:43.308 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:43.306204 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:43:43.322 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:43.320372 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2... 2022-03-02T06:43:43.349 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:43.346837 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-02T06:43:43.371 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:43.368843 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-02T06:43:44.796 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:43:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:43:45.312 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-02T06:43:45.312 INFO:tasks.rook: have 0/4 OSDs 2022-03-02T06:43:45.382 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:45.377793 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-02T06:43:51.414 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:51.412703 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10916 quincy" 2022-03-02T06:43:51.415 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:51.412724 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-02T06:43:51.415 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:51.412735 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10916 quincy", pursuing 2022-03-02T06:43:51.434 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:51.431636 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:43:51.572 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:51.570213 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:43:51.647 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:51.571254 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:43:52.252 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:52.250825 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10916 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2" 2022-03-02T06:43:52.411 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:52.409566 I | op-mon: start running mons 2022-03-02T06:43:52.430 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:52.428946 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:43:52.466 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:52.464383 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:43:52.472 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:52.470771 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:43:52.475 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:52.470962 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:43:53.373 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:53.371868 I | op-mon: targeting the mon count 1 2022-03-02T06:43:53.377 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:53.375609 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:43:53.722 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:53.720652 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:43:53.723 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:53.720703 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:43:54.054 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:54.052072 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:43:54.054 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:54.052102 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:43:54.398 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:54.395321 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:43:54.398 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:54.395354 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:43:54.781 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:54.779845 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:43:54.782 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:54.779879 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:43:55.123 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.121635 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:43:55.124 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.121671 I | op-config: deleting "log file" option from the mon configuration database 2022-03-02T06:43:55.313 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:43:55.528 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.526358 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-02T06:43:55.528 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.526379 I | op-mon: checking for basic quorum with existing mons 2022-03-02T06:43:55.597 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.596027 I | op-mon: mon "a" endpoint is [v2:10.103.147.119:3300,v1:10.103.147.119:6789] 2022-03-02T06:43:55.670 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.668910 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:43:55.685 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.684087 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:43:55.688 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.684334 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:43:55.816 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:43:55.834 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-02T06:43:55.835 INFO:tasks.rook: have 0/4 OSDs 2022-03-02T06:43:55.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.943707 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-02T06:43:55.954 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.953081 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-02T06:43:55.954 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:55.953120 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:43:56.153 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:56.151610 I | op-mon: mons running: [a] 2022-03-02T06:43:56.559 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:56.557305 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:43:56.559 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:56.557334 I | op-mon: mons created: 1 2022-03-02T06:43:56.967 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:56.965639 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:43:56.976 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:56.974770 I | op-mon: mons running: [a] 2022-03-02T06:43:57.400 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:57.399147 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:43:57.401 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:57.399185 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-02T06:43:57.815 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:57.813410 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-02T06:43:58.219 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:58.217352 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-02T06:43:58.631 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:58.629723 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-02T06:43:59.049 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:59.048221 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-02T06:43:59.050 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:59.048242 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-02T06:43:59.465 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:59.463365 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-02T06:43:59.886 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:59.884222 I | cephclient: successfully enabled msgr2 protocol 2022-03-02T06:43:59.886 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:59.884255 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-02T06:43:59.886 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:43:59.884267 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-02T06:44:00.293 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:00.291095 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-02T06:44:00.321 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:00.320002 I | op-mgr: start running mgr 2022-03-02T06:44:00.322 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:00.320025 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-02T06:44:00.786 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:00.785159 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-02T06:44:00.794 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:00.793590 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-02T06:44:00.798 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:00.796121 I | op-mgr: setting services to point to mgr "a" 2022-03-02T06:44:00.910 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:00.907780 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-02T06:44:00.911 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:00.907812 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-02T06:44:00.929 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:00.927930 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-02T06:44:01.047 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.045344 I | op-osd: creating 1 new PVCs for device set "smithi136-lv-1" 2022-03-02T06:44:01.240 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.238752 I | op-osd: successfully provisioned PVC "smithi136-lv-1-data-04lnm9" 2022-03-02T06:44:01.241 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.238783 I | op-osd: creating 1 new PVCs for device set "smithi136-lv-2" 2022-03-02T06:44:01.437 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.434858 I | op-osd: successfully provisioned PVC "smithi136-lv-2-data-0g889p" 2022-03-02T06:44:01.438 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.434889 I | op-osd: creating 1 new PVCs for device set "smithi136-lv-3" 2022-03-02T06:44:01.636 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.634565 I | op-osd: successfully provisioned PVC "smithi136-lv-3-data-0c469g" 2022-03-02T06:44:01.638 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.634654 I | op-osd: creating 1 new PVCs for device set "smithi136-lv-4" 2022-03-02T06:44:01.837 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.835697 I | op-osd: successfully provisioned PVC "smithi136-lv-4-data-07m48t" 2022-03-02T06:44:01.852 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.851098 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-1-data-04lnm9" 2022-03-02T06:44:01.895 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.893527 I | op-mgr: successful modules: dashboard 2022-03-02T06:44:01.899 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.898296 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-02T06:44:01.902 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:01.899434 I | op-mgr: successful modules: prometheus 2022-03-02T06:44:02.255 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:02.253367 I | op-mgr: successful modules: balancer 2022-03-02T06:44:02.271 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:02.269833 I | op-osd: started OSD provisioning job for PVC "smithi136-lv-1-data-04lnm9" 2022-03-02T06:44:02.271 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:02.269860 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-2-data-0g889p" 2022-03-02T06:44:02.655 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:02.653900 I | op-osd: started OSD provisioning job for PVC "smithi136-lv-2-data-0g889p" 2022-03-02T06:44:02.656 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:02.653927 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-3-data-0c469g" 2022-03-02T06:44:03.248 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:03.246638 I | op-osd: started OSD provisioning job for PVC "smithi136-lv-3-data-0c469g" 2022-03-02T06:44:03.248 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:03.246671 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-4-data-07m48t" 2022-03-02T06:44:03.252 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:03.250817 I | op-mgr: successful modules: orchestrator modules 2022-03-02T06:44:04.019 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:04.017876 I | op-osd: started OSD provisioning job for PVC "smithi136-lv-4-data-07m48t" 2022-03-02T06:44:04.019 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:04.017908 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-02T06:44:04.020 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:04.017916 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-02T06:44:04.065 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:04.063394 I | op-osd: OSD orchestration status for PVC smithi136-lv-1-data-04lnm9 is "starting" 2022-03-02T06:44:04.066 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:04.063554 I | op-osd: OSD orchestration status for PVC smithi136-lv-2-data-0g889p is "starting" 2022-03-02T06:44:04.067 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:04.063632 I | op-osd: OSD orchestration status for PVC smithi136-lv-3-data-0c469g is "starting" 2022-03-02T06:44:04.067 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:04.063718 I | op-osd: OSD orchestration status for PVC smithi136-lv-4-data-07m48t is "starting" 2022-03-02T06:44:05.836 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:44:06.811 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:44:06.829 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-02T06:44:06.830 INFO:tasks.rook: have 0/4 OSDs 2022-03-02T06:44:13.986 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:13.984615 I | op-osd: OSD orchestration status for node smithi136-lv-2-data-0g889p is "orchestrating" 2022-03-02T06:44:14.702 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:14.701149 I | op-osd: OSD orchestration status for PVC smithi136-lv-2-data-0g889p is "orchestrating" 2022-03-02T06:44:16.831 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:44:17.802 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:44:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":7,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1646203456,"num_remapped_pgs":0} 2022-03-02T06:44:17.823 INFO:tasks.rook: have 0/4 OSDs 2022-03-02T06:44:21.312 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:21.310803 I | op-osd: OSD orchestration status for PVC smithi136-lv-2-data-0g889p is "completed" 2022-03-02T06:44:21.312 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:21.311007 I | op-osd: creating OSD 0 on PVC "smithi136-lv-2-data-0g889p" 2022-03-02T06:44:21.312 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:21.311030 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-2-data-0g889p" 2022-03-02T06:44:23.421 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:23.418249 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi136": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T06:44:23.929 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:23.927963 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi136": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T06:44:23.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:23.947600 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi136": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T06:44:27.824 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:44:28.358 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:44:28.377 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":7,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1646203456,"num_remapped_pgs":0} 2022-03-02T06:44:28.378 INFO:tasks.rook: have 0/4 OSDs 2022-03-02T06:44:32.273 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:32.272724 I | op-osd: OSD orchestration status for node smithi136-lv-1-data-04lnm9 is "orchestrating" 2022-03-02T06:44:32.616 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:32.615349 I | op-osd: OSD orchestration status for node smithi136-lv-3-data-0c469g is "orchestrating" 2022-03-02T06:44:33.017 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:33.016368 I | op-osd: OSD orchestration status for PVC smithi136-lv-1-data-04lnm9 is "orchestrating" 2022-03-02T06:44:33.589 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:33.588464 I | op-osd: OSD orchestration status for node smithi136-lv-4-data-07m48t is "orchestrating" 2022-03-02T06:44:33.589 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:33.588538 I | op-osd: OSD orchestration status for PVC smithi136-lv-3-data-0c469g is "orchestrating" 2022-03-02T06:44:34.183 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:34.182484 I | op-osd: OSD orchestration status for PVC smithi136-lv-4-data-07m48t is "orchestrating" 2022-03-02T06:44:38.380 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:44:39.062 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:44:39.085 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646203475,"num_remapped_pgs":0} 2022-03-02T06:44:39.086 INFO:tasks.rook: have 0/4 OSDs 2022-03-02T06:44:39.321 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:39.318791 I | op-osd: OSD orchestration status for PVC smithi136-lv-1-data-04lnm9 is "completed" 2022-03-02T06:44:39.321 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:39.318810 I | op-osd: creating OSD 1 on PVC "smithi136-lv-1-data-04lnm9" 2022-03-02T06:44:39.321 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:39.318816 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-1-data-04lnm9" 2022-03-02T06:44:40.644 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:40.643857 I | op-osd: OSD orchestration status for PVC smithi136-lv-3-data-0c469g is "completed" 2022-03-02T06:44:40.644 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:40.643929 I | op-osd: creating OSD 2 on PVC "smithi136-lv-3-data-0c469g" 2022-03-02T06:44:40.645 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:40.643962 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-3-data-0c469g" 2022-03-02T06:44:41.808 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:41.807617 I | op-osd: OSD orchestration status for PVC smithi136-lv-4-data-07m48t is "completed" 2022-03-02T06:44:41.808 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:41.807653 I | op-osd: creating OSD 3 on PVC "smithi136-lv-4-data-07m48t" 2022-03-02T06:44:41.809 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:41.807669 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-4-data-07m48t" 2022-03-02T06:44:45.677 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:45.676913 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-02T06:44:45.678 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:44:45.676936 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:44:49.087 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:44:49.569 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:44:49.593 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646203475,"num_remapped_pgs":0} 2022-03-02T06:44:49.594 INFO:tasks.rook: have 0/4 OSDs 2022-03-02T06:44:59.595 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:45:00.087 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:00.108 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646203475,"num_remapped_pgs":0} 2022-03-02T06:45:00.108 INFO:tasks.rook: have 0/4 OSDs 2022-03-02T06:45:10.110 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:45:10.679 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:10.697 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":14,"num_osds":4,"num_up_osds":1,"osd_up_since":1646203502,"num_in_osds":4,"osd_in_since":1646203475,"num_remapped_pgs":0} 2022-03-02T06:45:10.698 INFO:tasks.rook: have 1/4 OSDs 2022-03-02T06:45:20.699 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd stat -f json 2022-03-02T06:45:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:21.233 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1646203517,"num_in_osds":4,"osd_in_since":1646203475,"num_remapped_pgs":0} 2022-03-02T06:45:21.233 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-02T06:45:21.234 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- cat /etc/ceph/ceph.conf 2022-03-02T06:45:21.396 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2022-03-02T06:45:21.397 INFO:teuthology.orchestra.run.smithi136.stdout:mon_host = 10.103.147.119:6789 2022-03-02T06:45:21.397 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:21.397 INFO:teuthology.orchestra.run.smithi136.stdout:[client.admin] 2022-03-02T06:45:21.398 INFO:teuthology.orchestra.run.smithi136.stdout:keyring = /etc/ceph/keyring 2022-03-02T06:45:21.401 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- cat /etc/ceph/keyring 2022-03-02T06:45:21.583 INFO:teuthology.orchestra.run.smithi136.stdout:[client.admin] 2022-03-02T06:45:21.584 INFO:teuthology.orchestra.run.smithi136.stdout:key = AQCtER9iMA6JNBAAqgsQ9Yyd/vFE3RJkf56DMA== 2022-03-02T06:45:21.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2022-03-02T06:45:21.598 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:45:21.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2022-03-02T06:45:21.653 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:45:21.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/keyring 2022-03-02T06:45:21.667 INFO:tasks.rook:Setting up client nodes... 2022-03-02T06:45:21.667 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-02T06:45:22.325 INFO:teuthology.orchestra.run.smithi136.stdout:[client.a] 2022-03-02T06:45:22.325 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCCEh9iVFsBDhAA4aayD8HugeyFqwWMTj0++g== 2022-03-02T06:45:22.345 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:45:22.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-02T06:45:22.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-02T06:45:22.367 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd dump --format=json 2022-03-02T06:45:22.937 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:22.938 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":20,"fsid":"4540ac50-27a7-4596-92d9-32f6a85efa3e","created":"2022-03-02T06:42:39.535854+0000","modified":"2022-03-02T06:45:21.700526+0000","last_up_change":"2022-03-02T06:45:17.980022+0000","last_in_change":"2022-03-02T06:44:35.821106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-02T06:45:19.433207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6104cb47-4733-4570-9c98-f76babd9f68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6800","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6801","nonce":3211239793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6802","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6803","nonce":3211239793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6806","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6807","nonce":3211239793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6804","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6805","nonce":3211239793}]},"public_addr":"10.252.56.22:6801/3211239793","cluster_addr":"10.252.56.22:6803/3211239793","heartbeat_back_addr":"10.252.56.22:6807/3211239793","heartbeat_front_addr":"10.252.56.22:6805/3211239793","state":["exists","up"]},{"osd":1,"uuid":"34b68fcf-df4f-4858-9632-88209ce6adfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6800","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6801","nonce":3025454178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6802","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6803","nonce":3025454178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6806","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6807","nonce":3025454178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6804","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6805","nonce":3025454178}]},"public_addr":"10.252.56.24:6801/3025454178","cluster_addr":"10.252.56.24:6803/3025454178","heartbeat_back_addr":"10.252.56.24:6807/3025454178","heartbeat_front_addr":"10.252.56.24:6805/3025454178","state":["exists","up"]},{"osd":2,"uuid":"4d6bdf36-ed2e-4019-8be8-48b21883d5f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6800","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6801","nonce":1129073201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6802","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6803","nonce":1129073201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6806","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6807","nonce":1129073201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6804","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6805","nonce":1129073201}]},"public_addr":"10.252.56.25:6801/1129073201","cluster_addr":"10.252.56.25:6803/1129073201","heartbeat_back_addr":"10.252.56.25:6807/1129073201","heartbeat_front_addr":"10.252.56.25:6805/1129073201","state":["exists","up"]},{"osd":3,"uuid":"39ce7b48-a4c8-42cf-8605-6634b937675e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6800","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6801","nonce":3630344479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6802","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6803","nonce":3630344479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6806","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6807","nonce":3630344479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6804","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6805","nonce":3630344479}]},"public_addr":"10.252.56.26:6801/3630344479","cluster_addr":"10.252.56.26:6803/3630344479","heartbeat_back_addr":"10.252.56.26:6807/3630344479","heartbeat_front_addr":"10.252.56.26:6805/3630344479","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:00.401444+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:15.900563+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:15.860691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:15.882103+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-02T06:45:22.979 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-02T06:45:19.433207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-02T06:45:22.979 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd pool get .mgr pg_num 2022-03-02T06:45:23.483 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2022-03-02T06:45:23.505 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-02T06:45:23.506 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-02T06:45:23.507 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph mgr dump --format=json 2022-03-02T06:45:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:24.061 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":15,"active_gid":4178,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.13:6800","nonce":1496400592},{"type":"v1","addr":"10.252.56.13:6801","nonce":1496400592}]},"active_addr":"10.252.56.13:6801/1496400592","active_change":"2022-03-02T06:43:17.975425+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.252.56.13:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.252.56.13:0","nonce":3196118973}]},{"addrvec":[{"type":"v2","addr":"10.252.56.13:0","nonce":904128631}]},{"addrvec":[{"type":"v2","addr":"10.252.56.13:0","nonce":674871725}]},{"addrvec":[{"type":"v2","addr":"10.252.56.13:0","nonce":1026421726}]}]}} 2022-03-02T06:45:24.066 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-02T06:45:24.066 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-02T06:45:24.067 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd dump --format=json 2022-03-02T06:45:24.556 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:24.557 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":20,"fsid":"4540ac50-27a7-4596-92d9-32f6a85efa3e","created":"2022-03-02T06:42:39.535854+0000","modified":"2022-03-02T06:45:21.700526+0000","last_up_change":"2022-03-02T06:45:17.980022+0000","last_in_change":"2022-03-02T06:44:35.821106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-02T06:45:19.433207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6104cb47-4733-4570-9c98-f76babd9f68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6800","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6801","nonce":3211239793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6802","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6803","nonce":3211239793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6806","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6807","nonce":3211239793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6804","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6805","nonce":3211239793}]},"public_addr":"10.252.56.22:6801/3211239793","cluster_addr":"10.252.56.22:6803/3211239793","heartbeat_back_addr":"10.252.56.22:6807/3211239793","heartbeat_front_addr":"10.252.56.22:6805/3211239793","state":["exists","up"]},{"osd":1,"uuid":"34b68fcf-df4f-4858-9632-88209ce6adfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6800","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6801","nonce":3025454178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6802","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6803","nonce":3025454178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6806","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6807","nonce":3025454178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6804","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6805","nonce":3025454178}]},"public_addr":"10.252.56.24:6801/3025454178","cluster_addr":"10.252.56.24:6803/3025454178","heartbeat_back_addr":"10.252.56.24:6807/3025454178","heartbeat_front_addr":"10.252.56.24:6805/3025454178","state":["exists","up"]},{"osd":2,"uuid":"4d6bdf36-ed2e-4019-8be8-48b21883d5f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6800","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6801","nonce":1129073201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6802","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6803","nonce":1129073201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6806","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6807","nonce":1129073201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6804","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6805","nonce":1129073201}]},"public_addr":"10.252.56.25:6801/1129073201","cluster_addr":"10.252.56.25:6803/1129073201","heartbeat_back_addr":"10.252.56.25:6807/1129073201","heartbeat_front_addr":"10.252.56.25:6805/1129073201","state":["exists","up"]},{"osd":3,"uuid":"39ce7b48-a4c8-42cf-8605-6634b937675e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6800","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6801","nonce":3630344479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6802","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6803","nonce":3630344479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6806","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6807","nonce":3630344479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6804","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6805","nonce":3630344479}]},"public_addr":"10.252.56.26:6801/3630344479","cluster_addr":"10.252.56.26:6803/3630344479","heartbeat_back_addr":"10.252.56.26:6807/3630344479","heartbeat_front_addr":"10.252.56.26:6805/3630344479","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:00.401444+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:15.900563+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:15.860691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:15.882103+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-02T06:45:24.588 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-02T06:45:24.588 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd dump --format=json 2022-03-02T06:45:25.134 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:25.134 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":20,"fsid":"4540ac50-27a7-4596-92d9-32f6a85efa3e","created":"2022-03-02T06:42:39.535854+0000","modified":"2022-03-02T06:45:21.700526+0000","last_up_change":"2022-03-02T06:45:17.980022+0000","last_in_change":"2022-03-02T06:44:35.821106+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-02T06:45:19.433207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6104cb47-4733-4570-9c98-f76babd9f68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6800","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6801","nonce":3211239793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6802","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6803","nonce":3211239793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6806","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6807","nonce":3211239793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.22:6804","nonce":3211239793},{"type":"v1","addr":"10.252.56.22:6805","nonce":3211239793}]},"public_addr":"10.252.56.22:6801/3211239793","cluster_addr":"10.252.56.22:6803/3211239793","heartbeat_back_addr":"10.252.56.22:6807/3211239793","heartbeat_front_addr":"10.252.56.22:6805/3211239793","state":["exists","up"]},{"osd":1,"uuid":"34b68fcf-df4f-4858-9632-88209ce6adfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6800","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6801","nonce":3025454178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6802","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6803","nonce":3025454178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6806","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6807","nonce":3025454178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.24:6804","nonce":3025454178},{"type":"v1","addr":"10.252.56.24:6805","nonce":3025454178}]},"public_addr":"10.252.56.24:6801/3025454178","cluster_addr":"10.252.56.24:6803/3025454178","heartbeat_back_addr":"10.252.56.24:6807/3025454178","heartbeat_front_addr":"10.252.56.24:6805/3025454178","state":["exists","up"]},{"osd":2,"uuid":"4d6bdf36-ed2e-4019-8be8-48b21883d5f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6800","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6801","nonce":1129073201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6802","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6803","nonce":1129073201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6806","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6807","nonce":1129073201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.25:6804","nonce":1129073201},{"type":"v1","addr":"10.252.56.25:6805","nonce":1129073201}]},"public_addr":"10.252.56.25:6801/1129073201","cluster_addr":"10.252.56.25:6803/1129073201","heartbeat_back_addr":"10.252.56.25:6807/1129073201","heartbeat_front_addr":"10.252.56.25:6805/1129073201","state":["exists","up"]},{"osd":3,"uuid":"39ce7b48-a4c8-42cf-8605-6634b937675e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6800","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6801","nonce":3630344479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6802","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6803","nonce":3630344479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6806","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6807","nonce":3630344479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.252.56.26:6804","nonce":3630344479},{"type":"v1","addr":"10.252.56.26:6805","nonce":3630344479}]},"public_addr":"10.252.56.26:6801/3630344479","cluster_addr":"10.252.56.26:6803/3630344479","heartbeat_back_addr":"10.252.56.26:6807/3630344479","heartbeat_front_addr":"10.252.56.26:6805/3630344479","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:00.401444+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:15.900563+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:15.860691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-02T06:45:15.882103+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-02T06:45:25.152 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph tell osd.0 flush_pg_stats 2022-03-02T06:45:25.153 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph tell osd.1 flush_pg_stats 2022-03-02T06:45:25.153 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph tell osd.2 flush_pg_stats 2022-03-02T06:45:25.154 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph tell osd.3 flush_pg_stats 2022-03-02T06:45:25.547 INFO:teuthology.orchestra.run.smithi136.stdout:55834574855 2022-03-02T06:45:25.547 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd last-stat-seq osd.0 2022-03-02T06:45:25.552 INFO:teuthology.orchestra.run.smithi136.stdout:73014444035 2022-03-02T06:45:25.552 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd last-stat-seq osd.1 2022-03-02T06:45:25.648 INFO:teuthology.orchestra.run.smithi136.stdout:73014444035 2022-03-02T06:45:25.649 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd last-stat-seq osd.3 2022-03-02T06:45:25.714 INFO:teuthology.orchestra.run.smithi136.stdout:73014444036 2022-03-02T06:45:25.715 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd last-stat-seq osd.2 2022-03-02T06:45:26.123 INFO:teuthology.orchestra.run.smithi136.stdout:73014444034 2022-03-02T06:45:26.143 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444035 got 73014444034 for osd.1 2022-03-02T06:45:26.192 INFO:teuthology.orchestra.run.smithi136.stdout:55834574854 2022-03-02T06:45:26.211 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2022-03-02T06:45:26.264 INFO:teuthology.orchestra.run.smithi136.stdout:73014444034 2022-03-02T06:45:26.283 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444035 got 73014444034 for osd.3 2022-03-02T06:45:26.285 INFO:teuthology.orchestra.run.smithi136.stdout:73014444035 2022-03-02T06:45:26.302 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444036 got 73014444035 for osd.2 2022-03-02T06:45:27.145 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd last-stat-seq osd.1 2022-03-02T06:45:27.213 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd last-stat-seq osd.0 2022-03-02T06:45:27.285 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd last-stat-seq osd.3 2022-03-02T06:45:27.303 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd last-stat-seq osd.2 2022-03-02T06:45:27.738 INFO:teuthology.orchestra.run.smithi136.stdout:55834574855 2022-03-02T06:45:27.757 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2022-03-02T06:45:27.758 DEBUG:teuthology.parallel:result is None 2022-03-02T06:45:27.772 INFO:teuthology.orchestra.run.smithi136.stdout:73014444035 2022-03-02T06:45:27.798 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444035 got 73014444035 for osd.1 2022-03-02T06:45:27.798 DEBUG:teuthology.parallel:result is None 2022-03-02T06:45:27.842 INFO:teuthology.orchestra.run.smithi136.stdout:73014444036 2022-03-02T06:45:27.861 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.2 2022-03-02T06:45:27.861 DEBUG:teuthology.parallel:result is None 2022-03-02T06:45:27.889 INFO:teuthology.orchestra.run.smithi136.stdout:73014444035 2022-03-02T06:45:27.911 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444035 got 73014444035 for osd.3 2022-03-02T06:45:27.912 DEBUG:teuthology.parallel:result is None 2022-03-02T06:45:27.912 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-02T06:45:27.912 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph pg dump --format=json 2022-03-02T06:45:28.379 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:28.382 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2022-03-02T06:45:28.398 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2022-03-02T06:45:26.920387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23016,"kb_used_data":1872,"kb_used_omap":0,"kb_used_meta":20224,"kb_avail":374957592,"statfs":{"total":383980142592,"available":383956574208,"internally_reserved":0,"allocated":1916928,"data_stored":1373988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20709376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000524"},"pg_stats":[{"pgid":"1.0","version":"20'85","reported_seq":122,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-03-02T06:45:22.626690+0000","last_change":"2022-03-02T06:45:21.463429+0000","last_active":"2022-03-02T06:45:22.626690+0000","last_peered":"2022-03-02T06:45:22.626690+0000","last_clean":"2022-03-02T06:45:22.626690+0000","last_became_active":"2022-03-02T06:45:21.462755+0000","last_became_peered":"2022-03-02T06:45:21.462755+0000","last_unstale":"2022-03-02T06:45:22.626690+0000","last_undegraded":"2022-03-02T06:45:22.626690+0000","last_fullsized":"2022-03-02T06:45:22.626690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-02T06:45:19.464290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-02T06:45:19.464290+0000","last_clean_scrub_stamp":"2022-03-02T06:45:19.464290+0000","objects_scrubbed":0,"log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-03T08:46:47.502474+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1290240,"data_stored":1277988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":4976,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740176,"statfs":{"total":95995035648,"available":95989940224,"internally_reserved":0,"allocated":172032,"data_stored":39360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6012,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739140,"statfs":{"total":95995035648,"available":95988879360,"internally_reserved":0,"allocated":581632,"data_stored":444876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6076,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739076,"statfs":{"total":95995035648,"available":95988813824,"internally_reserved":0,"allocated":581632,"data_stored":444876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5952,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739200,"statfs":{"total":95995035648,"available":95988940800,"internally_reserved":0,"allocated":581632,"data_stored":444876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-02T06:45:28.399 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph pg dump --format=json 2022-03-02T06:45:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:28.898 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2022-03-02T06:45:28.915 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2022-03-02T06:45:26.920387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23016,"kb_used_data":1872,"kb_used_omap":0,"kb_used_meta":20224,"kb_avail":374957592,"statfs":{"total":383980142592,"available":383956574208,"internally_reserved":0,"allocated":1916928,"data_stored":1373988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20709376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000524"},"pg_stats":[{"pgid":"1.0","version":"20'85","reported_seq":122,"reported_epoch":20,"state":"active+clean","last_fresh":"2022-03-02T06:45:22.626690+0000","last_change":"2022-03-02T06:45:21.463429+0000","last_active":"2022-03-02T06:45:22.626690+0000","last_peered":"2022-03-02T06:45:22.626690+0000","last_clean":"2022-03-02T06:45:22.626690+0000","last_became_active":"2022-03-02T06:45:21.462755+0000","last_became_peered":"2022-03-02T06:45:21.462755+0000","last_unstale":"2022-03-02T06:45:22.626690+0000","last_undegraded":"2022-03-02T06:45:22.626690+0000","last_fullsized":"2022-03-02T06:45:22.626690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-02T06:45:19.464290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-02T06:45:19.464290+0000","last_clean_scrub_stamp":"2022-03-02T06:45:19.464290+0000","objects_scrubbed":0,"log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-03T08:46:47.502474+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1290240,"data_stored":1277988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":17,"seq":73014444035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":4976,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740176,"statfs":{"total":95995035648,"available":95989940224,"internally_reserved":0,"allocated":172032,"data_stored":39360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6012,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739140,"statfs":{"total":95995035648,"available":95988879360,"internally_reserved":0,"allocated":581632,"data_stored":444876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6076,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739076,"statfs":{"total":95995035648,"available":95988813824,"internally_reserved":0,"allocated":581632,"data_stored":444876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]}]},{"osd":1,"up_from":17,"seq":73014444035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5952,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739200,"statfs":{"total":95995035648,"available":95988940800,"internally_reserved":0,"allocated":581632,"data_stored":444876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5289999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-02T06:45:28.915 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-02T06:45:28.916 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-02T06:45:28.916 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-02T06:45:28.917 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph health --format=json 2022-03-02T06:45:29.454 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:29.454 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-02T06:45:29.472 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-02T06:45:29.472 INFO:tasks.rook:Rook complete, yielding 2022-03-02T06:45:29.473 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-02T06:45:29.486 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph -s 2022-03-02T06:45:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: cluster: 2022-03-02T06:45:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: id: 4540ac50-27a7-4596-92d9-32f6a85efa3e 2022-03-02T06:45:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: health: HEALTH_OK 2022-03-02T06:45:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: services: 2022-03-02T06:45:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: mon: 1 daemons, quorum a (age 2m) 2022-03-02T06:45:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: mgr: a(active, since 2m) 2022-03-02T06:45:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 54s) 2022-03-02T06:45:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: data: 2022-03-02T06:45:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: pools: 1 pools, 1 pgs 2022-03-02T06:45:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: objects: 2 objects, 577 KiB 2022-03-02T06:45:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: usage: 22 MiB used, 358 GiB / 358 GiB avail 2022-03-02T06:45:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: pgs: 1 active+clean 2022-03-02T06:45:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T06:45:30.100 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch status 2022-03-02T06:45:30.725 INFO:teuthology.orchestra.run.smithi136.stdout:Backend: rook 2022-03-02T06:45:30.725 INFO:teuthology.orchestra.run.smithi136.stdout:Available: Yes 2022-03-02T06:45:30.749 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ps 2022-03-02T06:45:31.278 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-02T06:45:31.278 INFO:teuthology.orchestra.run.smithi136.stdout:crashcollector.crash smithi136 running (68s) 0s ago 69s - - 623b85caee6c 2022-03-02T06:45:31.278 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.a smithi136 running (2m) 0s ago 2m - - 623b85caee6c 2022-03-02T06:45:31.278 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (3m) 0s ago 3m - - 623b85caee6c 2022-03-02T06:45:31.279 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (69s) 0s ago 70s - - 623b85caee6c 2022-03-02T06:45:31.279 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (50s) 0s ago 50s - - 623b85caee6c 2022-03-02T06:45:31.279 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (49s) 0s ago 49s - - 623b85caee6c 2022-03-02T06:45:31.279 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (47s) 0s ago 48s - - 623b85caee6c 2022-03-02T06:45:31.296 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls 2022-03-02T06:45:31.908 INFO:teuthology.orchestra.run.smithi136.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-02T06:45:31.908 INFO:teuthology.orchestra.run.smithi136.stdout:crash 1/1 0s ago 69s * 2022-03-02T06:45:31.908 INFO:teuthology.orchestra.run.smithi136.stdout:mgr 1/1 0s ago 2m count:1 2022-03-02T06:45:31.908 INFO:teuthology.orchestra.run.smithi136.stdout:mon 1/1 0s ago 3m count:1 2022-03-02T06:45:31.909 INFO:teuthology.orchestra.run.smithi136.stdout:osd 8 0s ago 70s 2022-03-02T06:45:31.909 INFO:teuthology.orchestra.run.smithi136.stdout:osd.all-available-devices 0 0s ago - * 2022-03-02T06:45:31.932 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch device ls 2022-03-02T06:45:32.478 INFO:teuthology.orchestra.run.smithi136.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-02T06:45:32.479 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-02T06:45:32.479 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-02T06:45:32.479 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-02T06:45:32.479 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-02T06:45:32.502 INFO:teuthology.run_tasks:Running task install... 2022-03-02T06:45:32.515 DEBUG:teuthology.task.install:project ceph 2022-03-02T06:45:32.515 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2'}} 2022-03-02T06:45:32.516 DEBUG:teuthology.task.install:config {'host.a': None, 'sha1': '8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2'} 2022-03-02T06:45:32.516 INFO:teuthology.task.install:Using flavor: default 2022-03-02T06:45:32.523 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-02T06:45:32.523 INFO:teuthology.task.install:extra packages: [] 2022-03-02T06:45:32.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo lsb_release -is 2022-03-02T06:45:32.563 INFO:teuthology.orchestra.run.smithi136.stdout:Ubuntu 2022-03-02T06:45:32.568 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-02T06:45:32.568 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-key list | grep Ceph 2022-03-02T06:45:32.648 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-03-02T06:45:32.706 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-03-02T06:45:32.706 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph.com (release key) 2022-03-02T06:45:32.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo lsb_release -is 2022-03-02T06:45:32.755 INFO:teuthology.orchestra.run.smithi136.stdout:Ubuntu 2022-03-02T06:45:32.761 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-02T06:45:32.762 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-03-02T06:45:32.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 2022-03-02T06:45:32.992 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default/ 2022-03-02T06:45:32.992 INFO:teuthology.task.install.deb:Package version is 17.0.0-10916-g8b0a50f4-1focal 2022-03-02T06:45:33.163 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:45:33.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-03-02T06:45:33.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2022-03-02T06:45:33.328 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-02T06:45:33.442 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal InRelease 2022-03-02T06:45:33.442 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-02T06:45:33.478 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-03-02T06:45:33.517 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal Release.gpg 2022-03-02T06:45:33.525 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-02T06:45:33.534 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease 2022-03-02T06:45:33.553 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2022-03-02T06:45:33.599 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main i386 Packages [3,357 B] 2022-03-02T06:45:33.607 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:10 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-02T06:45:34.015 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 28.4 kB in 1s (44.2 kB/s) 2022-03-02T06:45:34.936 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:45:34.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-10916-g8b0a50f4-1focal cephadm=17.0.0-10916-g8b0a50f4-1focal ceph-mds=17.0.0-10916-g8b0a50f4-1focal ceph-mgr=17.0.0-10916-g8b0a50f4-1focal ceph-common=17.0.0-10916-g8b0a50f4-1focal ceph-fuse=17.0.0-10916-g8b0a50f4-1focal ceph-test=17.0.0-10916-g8b0a50f4-1focal ceph-volume=17.0.0-10916-g8b0a50f4-1focal radosgw=17.0.0-10916-g8b0a50f4-1focal python3-rados=17.0.0-10916-g8b0a50f4-1focal python3-rgw=17.0.0-10916-g8b0a50f4-1focal python3-cephfs=17.0.0-10916-g8b0a50f4-1focal python3-rbd=17.0.0-10916-g8b0a50f4-1focal libcephfs2=17.0.0-10916-g8b0a50f4-1focal libcephfs-dev=17.0.0-10916-g8b0a50f4-1focal librados2=17.0.0-10916-g8b0a50f4-1focal librbd1=17.0.0-10916-g8b0a50f4-1focal rbd-fuse=17.0.0-10916-g8b0a50f4-1focal 2022-03-02T06:45:35.029 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2022-03-02T06:45:35.211 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2022-03-02T06:45:35.212 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2022-03-02T06:45:35.381 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2022-03-02T06:45:35.382 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-03-02T06:45:35.383 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2022-03-02T06:45:35.383 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2022-03-02T06:45:35.384 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-03-02T06:45:35.384 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2022-03-02T06:45:35.385 INFO:teuthology.orchestra.run.smithi136.stdout: libdouble-conversion3 libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-03-02T06:45:35.385 INFO:teuthology.orchestra.run.smithi136.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2022-03-02T06:45:35.385 INFO:teuthology.orchestra.run.smithi136.stdout: libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-02T06:45:35.386 INFO:teuthology.orchestra.run.smithi136.stdout: libsqlite3-mod-ceph libthrift-0.13.0 libxslt1.1 nvme-cli 2022-03-02T06:45:35.386 INFO:teuthology.orchestra.run.smithi136.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2022-03-02T06:45:35.387 INFO:teuthology.orchestra.run.smithi136.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2022-03-02T06:45:35.387 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2022-03-02T06:45:35.388 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth python3-html5lib python3-joblib python3-jwt 2022-03-02T06:45:35.388 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2022-03-02T06:45:35.389 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-03-02T06:45:35.389 INFO:teuthology.orchestra.run.smithi136.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-03-02T06:45:35.389 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-02T06:45:35.391 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-03-02T06:45:35.392 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-03-02T06:45:35.392 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-03-02T06:45:35.392 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-03-02T06:45:35.392 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-03-02T06:45:35.393 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2022-03-02T06:45:35.393 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet 2022-03-02T06:45:35.393 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2022-03-02T06:45:35.394 INFO:teuthology.orchestra.run.smithi136.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-03-02T06:45:35.394 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-vectors python3-genshi python3-crypto python3-lxml-dbg 2022-03-02T06:45:35.394 INFO:teuthology.orchestra.run.smithi136.stdout: python-lxml-doc python3-beaker python-mako-doc python-openssl-doc 2022-03-02T06:45:35.394 INFO:teuthology.orchestra.run.smithi136.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2022-03-02T06:45:35.395 INFO:teuthology.orchestra.run.smithi136.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2022-03-02T06:45:35.395 INFO:teuthology.orchestra.run.smithi136.stdout: python-pyinotify-doc python-scipy-doc python3-dap python-sklearn-doc 2022-03-02T06:45:35.395 INFO:teuthology.orchestra.run.smithi136.stdout: ipython3 python-waitress-doc python-webob-doc python-webtest-doc 2022-03-02T06:45:35.395 INFO:teuthology.orchestra.run.smithi136.stdout: python-werkzeug-doc python3-termcolor python3-watchdog 2022-03-02T06:45:35.396 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2022-03-02T06:45:35.396 INFO:teuthology.orchestra.run.smithi136.stdout: btrfs-tools ceph-mgr-cephadm 2022-03-02T06:45:35.501 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2022-03-02T06:45:35.501 INFO:teuthology.orchestra.run.smithi136.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-dashboard 2022-03-02T06:45:35.501 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-03-02T06:45:35.502 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon ceph-osd ceph-test ceph-volume cephadm jq libcephfs-dev libcephfs2 2022-03-02T06:45:35.502 INFO:teuthology.orchestra.run.smithi136.stdout: libdouble-conversion3 libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-03-02T06:45:35.502 INFO:teuthology.orchestra.run.smithi136.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2022-03-02T06:45:35.504 INFO:teuthology.orchestra.run.smithi136.stdout: libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-02T06:45:35.504 INFO:teuthology.orchestra.run.smithi136.stdout: libsqlite3-mod-ceph libthrift-0.13.0 libxslt1.1 nvme-cli 2022-03-02T06:45:35.504 INFO:teuthology.orchestra.run.smithi136.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2022-03-02T06:45:35.505 INFO:teuthology.orchestra.run.smithi136.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2022-03-02T06:45:35.505 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2022-03-02T06:45:35.505 INFO:teuthology.orchestra.run.smithi136.stdout: python3-decorator python3-google-auth python3-html5lib python3-joblib 2022-03-02T06:45:35.505 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-03-02T06:45:35.506 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-03-02T06:45:35.506 INFO:teuthology.orchestra.run.smithi136.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-03-02T06:45:35.506 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-02T06:45:35.506 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2022-03-02T06:45:35.507 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2022-03-02T06:45:35.508 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-03-02T06:45:35.508 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-03-02T06:45:35.508 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-03-02T06:45:35.508 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-03-02T06:45:35.509 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse 2022-03-02T06:45:35.509 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet 2022-03-02T06:45:35.509 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages will be upgraded: 2022-03-02T06:45:35.509 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 librbd1 2022-03-02T06:45:35.697 INFO:teuthology.orchestra.run.smithi136.stdout:2 upgraded, 102 newly installed, 0 to remove and 65 not upgraded. 2022-03-02T06:45:35.698 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 186 MB of archives. 2022-03-02T06:45:35.698 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 869 MB of additional disk space will be used. 2022-03-02T06:45:35.698 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-03-02T06:45:35.730 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-10916-g8b0a50f4-1focal [3,484 kB] 2022-03-02T06:45:36.082 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2022-03-02T06:45:36.091 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libpcre2-16-0 amd64 10.34-7 [181 kB] 2022-03-02T06:45:36.177 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2022-03-02T06:45:36.416 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2022-03-02T06:45:36.426 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2022-03-02T06:45:36.452 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-10916-g8b0a50f4-1focal [3,687 kB] 2022-03-02T06:45:36.456 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2022-03-02T06:45:36.460 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-03-02T06:45:36.461 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-03-02T06:45:36.466 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-03-02T06:45:36.466 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-03-02T06:45:36.481 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-03-02T06:45:36.482 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-03-02T06:45:36.499 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-03-02T06:45:36.570 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-03-02T06:45:36.571 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-03-02T06:45:36.572 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-03-02T06:45:36.573 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-03-02T06:45:36.573 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-03-02T06:45:36.574 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-03-02T06:45:36.575 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-03-02T06:45:36.575 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-03-02T06:45:36.581 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-03-02T06:45:36.658 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-03-02T06:45:36.659 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-03-02T06:45:36.661 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2022-03-02T06:45:36.663 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-03-02T06:45:36.664 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-03-02T06:45:36.670 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-03-02T06:45:36.671 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-03-02T06:45:36.671 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-03-02T06:45:36.672 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-03-02T06:45:36.751 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-03-02T06:45:36.755 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-03-02T06:45:36.756 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-03-02T06:45:36.759 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-03-02T06:45:36.761 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-03-02T06:45:36.788 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-10916-g8b0a50f4-1focal [752 kB] 2022-03-02T06:45:36.833 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-03-02T06:45:36.834 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-03-02T06:45:36.835 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-03-02T06:45:36.856 INFO:teuthology.orchestra.run.smithi136.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-10916-g8b0a50f4-1focal [413 kB] 2022-03-02T06:45:36.888 INFO:teuthology.orchestra.run.smithi136.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-10916-g8b0a50f4-1focal [22.4 kB] 2022-03-02T06:45:36.889 INFO:teuthology.orchestra.run.smithi136.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-10916-g8b0a50f4-1focal [200 kB] 2022-03-02T06:45:36.893 INFO:teuthology.orchestra.run.smithi136.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-10916-g8b0a50f4-1focal [41.9 kB] 2022-03-02T06:45:36.895 INFO:teuthology.orchestra.run.smithi136.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-10916-g8b0a50f4-1focal [385 kB] 2022-03-02T06:45:36.959 INFO:teuthology.orchestra.run.smithi136.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-10916-g8b0a50f4-1focal [4,160 kB] 2022-03-02T06:45:37.148 INFO:teuthology.orchestra.run.smithi136.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-03-02T06:45:37.192 INFO:teuthology.orchestra.run.smithi136.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-10916-g8b0a50f4-1focal [116 kB] 2022-03-02T06:45:37.193 INFO:teuthology.orchestra.run.smithi136.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-10916-g8b0a50f4-1focal [478 kB] 2022-03-02T06:45:37.223 INFO:teuthology.orchestra.run.smithi136.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-03-02T06:45:37.291 INFO:teuthology.orchestra.run.smithi136.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-03-02T06:45:37.291 INFO:teuthology.orchestra.run.smithi136.stdout:Get:55 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-10916-g8b0a50f4-1focal [24.4 MB] 2022-03-02T06:45:37.378 INFO:teuthology.orchestra.run.smithi136.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-03-02T06:45:37.379 INFO:teuthology.orchestra.run.smithi136.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-03-02T06:45:37.379 INFO:teuthology.orchestra.run.smithi136.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-03-02T06:45:37.380 INFO:teuthology.orchestra.run.smithi136.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-03-02T06:45:37.380 INFO:teuthology.orchestra.run.smithi136.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-03-02T06:45:37.381 INFO:teuthology.orchestra.run.smithi136.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-03-02T06:45:37.381 INFO:teuthology.orchestra.run.smithi136.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-03-02T06:45:37.382 INFO:teuthology.orchestra.run.smithi136.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-03-02T06:45:37.399 INFO:teuthology.orchestra.run.smithi136.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-03-02T06:45:37.467 INFO:teuthology.orchestra.run.smithi136.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-03-02T06:45:37.474 INFO:teuthology.orchestra.run.smithi136.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-03-02T06:45:37.474 INFO:teuthology.orchestra.run.smithi136.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-03-02T06:45:37.475 INFO:teuthology.orchestra.run.smithi136.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-03-02T06:45:37.475 INFO:teuthology.orchestra.run.smithi136.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-03-02T06:45:37.477 INFO:teuthology.orchestra.run.smithi136.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-03-02T06:45:37.479 INFO:teuthology.orchestra.run.smithi136.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-03-02T06:45:37.483 INFO:teuthology.orchestra.run.smithi136.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-03-02T06:45:37.485 INFO:teuthology.orchestra.run.smithi136.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-03-02T06:45:37.569 INFO:teuthology.orchestra.run.smithi136.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-03-02T06:45:37.570 INFO:teuthology.orchestra.run.smithi136.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-03-02T06:45:37.572 INFO:teuthology.orchestra.run.smithi136.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-03-02T06:45:37.592 INFO:teuthology.orchestra.run.smithi136.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-03-02T06:45:37.592 INFO:teuthology.orchestra.run.smithi136.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-03-02T06:45:37.593 INFO:teuthology.orchestra.run.smithi136.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-03-02T06:45:37.593 INFO:teuthology.orchestra.run.smithi136.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-03-02T06:45:37.594 INFO:teuthology.orchestra.run.smithi136.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-03-02T06:45:37.594 INFO:teuthology.orchestra.run.smithi136.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-03-02T06:45:37.653 INFO:teuthology.orchestra.run.smithi136.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-03-02T06:45:37.735 INFO:teuthology.orchestra.run.smithi136.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-03-02T06:45:37.738 INFO:teuthology.orchestra.run.smithi136.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-03-02T06:45:37.740 INFO:teuthology.orchestra.run.smithi136.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2022-03-02T06:45:38.246 INFO:teuthology.orchestra.run.smithi136.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-10916-g8b0a50f4-1focal [6,423 kB] 2022-03-02T06:45:38.366 INFO:teuthology.orchestra.run.smithi136.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-10916-g8b0a50f4-1focal [210 kB] 2022-03-02T06:45:38.373 INFO:teuthology.orchestra.run.smithi136.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-10916-g8b0a50f4-1focal [151 kB] 2022-03-02T06:45:38.376 INFO:teuthology.orchestra.run.smithi136.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-10916-g8b0a50f4-1focal [1,571 kB] 2022-03-02T06:45:38.429 INFO:teuthology.orchestra.run.smithi136.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-10916-g8b0a50f4-1focal [7,811 kB] 2022-03-02T06:45:38.761 INFO:teuthology.orchestra.run.smithi136.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-10916-g8b0a50f4-1focal [28.7 MB] 2022-03-02T06:45:39.839 INFO:teuthology.orchestra.run.smithi136.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-10916-g8b0a50f4-1focal [3,808 B] 2022-03-02T06:45:39.839 INFO:teuthology.orchestra.run.smithi136.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-10916-g8b0a50f4-1focal [860 kB] 2022-03-02T06:45:39.847 INFO:teuthology.orchestra.run.smithi136.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-10916-g8b0a50f4-1focal [2,385 kB] 2022-03-02T06:45:39.867 INFO:teuthology.orchestra.run.smithi136.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-10916-g8b0a50f4-1focal [1,677 kB] 2022-03-02T06:45:39.935 INFO:teuthology.orchestra.run.smithi136.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-10916-g8b0a50f4-1focal [7,887 kB] 2022-03-02T06:45:40.267 INFO:teuthology.orchestra.run.smithi136.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-10916-g8b0a50f4-1focal [3,928 B] 2022-03-02T06:45:40.268 INFO:teuthology.orchestra.run.smithi136.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-10916-g8b0a50f4-1focal [50.5 MB] 2022-03-02T06:45:42.272 INFO:teuthology.orchestra.run.smithi136.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-10916-g8b0a50f4-1focal [219 kB] 2022-03-02T06:45:42.274 INFO:teuthology.orchestra.run.smithi136.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-10916-g8b0a50f4-1focal [69.5 kB] 2022-03-02T06:45:42.274 INFO:teuthology.orchestra.run.smithi136.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-10916-g8b0a50f4-1focal [21.4 kB] 2022-03-02T06:45:42.275 INFO:teuthology.orchestra.run.smithi136.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-10916-g8b0a50f4-1focal [12.7 MB] 2022-03-02T06:45:42.653 INFO:teuthology.orchestra.run.smithi136.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-10916-g8b0a50f4-1focal [79.1 kB] 2022-03-02T06:45:43.292 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 186 MB in 7s (26.1 MB/s) 2022-03-02T06:45:43.495 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-03-02T06:45:43.544 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109337 files and directories currently installed.) 2022-03-02T06:45:43.548 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-03-02T06:45:43.795 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-02T06:45:44.282 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2022-03-02T06:45:44.297 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2022-03-02T06:45:44.342 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2022-03-02T06:45:44.693 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2022-03-02T06:45:44.704 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7_amd64.deb ... 2022-03-02T06:45:44.753 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7) ... 2022-03-02T06:45:45.323 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5core5a:amd64. 2022-03-02T06:45:45.335 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-03-02T06:45:45.378 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-03-02T06:45:46.077 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2022-03-02T06:45:46.092 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-03-02T06:45:46.160 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-03-02T06:45:46.522 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5network5:amd64. 2022-03-02T06:45:46.535 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-03-02T06:45:46.588 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-03-02T06:45:46.991 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libthrift-0.13.0. 2022-03-02T06:45:47.005 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2022-03-02T06:45:47.050 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2022-03-02T06:45:47.616 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../007-librbd1_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:47.737 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librbd1 (17.0.0-10916-g8b0a50f4-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-02T06:45:48.520 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../008-librados2_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:48.744 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librados2 (17.0.0-10916-g8b0a50f4-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-02T06:45:49.530 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs2. 2022-03-02T06:45:49.542 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../009-libcephfs2_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:49.580 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs2 (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:50.093 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rados. 2022-03-02T06:45:50.106 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../010-python3-rados_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:50.167 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rados (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:50.554 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-03-02T06:45:50.568 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../011-python3-ceph-argparse_17.0.0-10916-g8b0a50f4-1focal_all.deb ... 2022-03-02T06:45:50.612 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-argparse (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:50.906 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cephfs. 2022-03-02T06:45:50.918 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../012-python3-cephfs_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:50.964 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cephfs (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:51.360 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-common. 2022-03-02T06:45:51.372 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../013-python3-ceph-common_17.0.0-10916-g8b0a50f4-1focal_all.deb ... 2022-03-02T06:45:51.434 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-common (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:51.779 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-prettytable. 2022-03-02T06:45:51.792 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2022-03-02T06:45:51.879 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-03-02T06:45:52.293 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rbd. 2022-03-02T06:45:52.307 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../015-python3-rbd_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:52.343 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rbd (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:52.805 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-03-02T06:45:52.822 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-03-02T06:45:52.863 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-02T06:45:53.251 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-03-02T06:45:53.263 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2022-03-02T06:45:53.314 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-03-02T06:45:53.838 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-03-02T06:45:53.854 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-03-02T06:45:53.980 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-02T06:45:54.401 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librgw2. 2022-03-02T06:45:54.415 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../019-librgw2_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:54.459 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librgw2 (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:55.305 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rgw. 2022-03-02T06:45:55.320 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../020-python3-rgw_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:55.392 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rgw (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:55.800 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liboath0. 2022-03-02T06:45:55.814 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2022-03-02T06:45:55.850 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-03-02T06:45:56.281 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libradosstriper1. 2022-03-02T06:45:56.294 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../022-libradosstriper1_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:56.355 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libradosstriper1 (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:56.858 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-common. 2022-03-02T06:45:56.872 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../023-ceph-common_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:56.941 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-common (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:45:59.668 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-base. 2022-03-02T06:45:59.685 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../024-ceph-base_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:45:59.733 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-base (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:00.800 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cryptography. 2022-03-02T06:46:00.812 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-03-02T06:46:00.857 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-02T06:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-openssl. 2022-03-02T06:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2022-03-02T06:46:01.353 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-03-02T06:46:01.665 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-03-02T06:46:01.673 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-10916-g8b0a50f4-1focal_all.deb ... 2022-03-02T06:46:01.716 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-modules-core (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:02.314 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-bcrypt. 2022-03-02T06:46:02.326 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-03-02T06:46:02.355 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-02T06:46:02.759 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cherrypy3. 2022-03-02T06:46:02.773 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2022-03-02T06:46:02.815 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-03-02T06:46:03.214 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jwt. 2022-03-02T06:46:03.227 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../030-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-03-02T06:46:03.313 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-03-02T06:46:03.791 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-logutils. 2022-03-02T06:46:03.804 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-7_all.deb ... 2022-03-02T06:46:03.934 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-03-02T06:46:04.348 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-markupsafe. 2022-03-02T06:46:04.356 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../032-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-03-02T06:46:04.398 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-03-02T06:46:04.718 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-mako. 2022-03-02T06:46:04.729 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../033-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-03-02T06:46:04.805 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-02T06:46:05.307 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplegeneric. 2022-03-02T06:46:05.323 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-3_all.deb ... 2022-03-02T06:46:05.365 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-03-02T06:46:05.766 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-singledispatch. 2022-03-02T06:46:05.776 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-03-02T06:46:05.819 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-03-02T06:46:06.250 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webob. 2022-03-02T06:46:06.261 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../036-python3-webob_1%3a1.8.5-2_all.deb ... 2022-03-02T06:46:06.356 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-03-02T06:46:06.718 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-soupsieve. 2022-03-02T06:46:06.727 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../037-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-03-02T06:46:06.777 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-02T06:46:07.038 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-bs4. 2022-03-02T06:46:07.047 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../038-python3-bs4_4.8.2-1_all.deb ... 2022-03-02T06:46:07.129 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-03-02T06:46:08.424 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-waitress. 2022-03-02T06:46:08.440 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../039-python3-waitress_1.4.1-1_all.deb ... 2022-03-02T06:46:08.555 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-waitress (1.4.1-1) ... 2022-03-02T06:46:09.007 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempita. 2022-03-02T06:46:09.019 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../040-python3-tempita_0.5.2-6_all.deb ... 2022-03-02T06:46:09.098 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-03-02T06:46:09.502 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-paste. 2022-03-02T06:46:09.511 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../041-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-03-02T06:46:09.569 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-03-02T06:46:09.949 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-03-02T06:46:09.961 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../042-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-03-02T06:46:09.998 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-03-02T06:46:10.360 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastedeploy. 2022-03-02T06:46:10.376 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../043-python3-pastedeploy_2.1.0-1_all.deb ... 2022-03-02T06:46:10.419 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-03-02T06:46:10.873 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webtest. 2022-03-02T06:46:10.882 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../044-python3-webtest_2.0.32-2_all.deb ... 2022-03-02T06:46:10.922 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-03-02T06:46:11.243 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pecan. 2022-03-02T06:46:11.255 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../045-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-03-02T06:46:11.292 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-03-02T06:46:11.794 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-werkzeug. 2022-03-02T06:46:11.804 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../046-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-03-02T06:46:11.884 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-02T06:46:12.347 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-03-02T06:46:12.357 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../047-libsqlite3-mod-ceph_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:12.419 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:12.793 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr. 2022-03-02T06:46:12.806 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../048-ceph-mgr_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:12.843 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:14.349 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mon. 2022-03-02T06:46:14.360 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../049-ceph-mon_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:14.441 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mon (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:15.663 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-osd. 2022-03-02T06:46:15.675 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../050-ceph-osd_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:15.712 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-osd (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:18.713 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph. 2022-03-02T06:46:18.728 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../051-ceph_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:18.796 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:19.269 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-fuse. 2022-03-02T06:46:19.279 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../052-ceph-fuse_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:19.344 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-fuse (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:19.858 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mds. 2022-03-02T06:46:19.869 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../053-ceph-mds_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:19.908 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mds (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:20.621 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-repoze.lru. 2022-03-02T06:46:20.634 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-2_all.deb ... 2022-03-02T06:46:20.671 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-03-02T06:46:20.974 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjs-underscore. 2022-03-02T06:46:20.987 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../055-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-03-02T06:46:21.033 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-02T06:46:21.394 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-03-02T06:46:21.405 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../056-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-03-02T06:46:21.478 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-02T06:46:21.790 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-routes. 2022-03-02T06:46:21.799 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../057-python3-routes_2.4.1-2_all.deb ... 2022-03-02T06:46:21.864 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-03-02T06:46:22.269 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-03-02T06:46:22.283 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../058-ceph-mgr-dashboard_17.0.0-10916-g8b0a50f4-1focal_all.deb ... 2022-03-02T06:46:22.335 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-dashboard (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:22.934 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-decorator. 2022-03-02T06:46:22.943 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-03-02T06:46:23.041 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-03-02T06:46:23.387 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-03-02T06:46:23.400 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-03-02T06:46:23.445 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-02T06:46:23.874 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-scipy. 2022-03-02T06:46:23.889 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-03-02T06:46:23.940 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-03-02T06:46:25.800 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-03-02T06:46:25.812 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-03-02T06:46:25.854 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-02T06:46:26.443 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-joblib. 2022-03-02T06:46:26.455 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-03-02T06:46:26.576 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-03-02T06:46:27.047 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn. 2022-03-02T06:46:27.059 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-03-02T06:46:27.097 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-02T06:46:27.951 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-03-02T06:46:27.967 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_17.0.0-10916-g8b0a50f4-1focal_all.deb ... 2022-03-02T06:46:28.046 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:29.455 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cachetools. 2022-03-02T06:46:29.468 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-03-02T06:46:29.516 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-03-02T06:46:30.101 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1. 2022-03-02T06:46:30.118 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-03-02T06:46:30.161 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-03-02T06:46:30.567 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-03-02T06:46:30.580 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-03-02T06:46:30.628 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-02T06:46:31.080 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rsa. 2022-03-02T06:46:31.094 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-03-02T06:46:31.153 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rsa (4.0-3) ... 2022-03-02T06:46:31.527 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-google-auth. 2022-03-02T06:46:31.539 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-03-02T06:46:31.628 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-03-02T06:46:31.973 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-blinker. 2022-03-02T06:46:31.982 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-03-02T06:46:32.044 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-02T06:46:32.489 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-oauthlib. 2022-03-02T06:46:32.501 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-03-02T06:46:32.550 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-02T06:46:32.954 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-03-02T06:46:32.965 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-03-02T06:46:33.008 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-02T06:46:33.614 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-websocket. 2022-03-02T06:46:33.623 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-03-02T06:46:33.663 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-03-02T06:46:34.013 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-kubernetes. 2022-03-02T06:46:34.024 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-03-02T06:46:34.063 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-03-02T06:46:34.890 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-03-02T06:46:34.902 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-10916-g8b0a50f4-1focal_all.deb ... 2022-03-02T06:46:34.978 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:35.517 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libonig5:amd64. 2022-03-02T06:46:35.531 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-03-02T06:46:35.584 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-03-02T06:46:36.001 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjq1:amd64. 2022-03-02T06:46:36.012 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-02T06:46:36.051 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-02T06:46:36.366 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package jq. 2022-03-02T06:46:36.379 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-02T06:46:36.415 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-03-02T06:46:36.924 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-03-02T06:46:36.938 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../080-libxslt1.1_1.1.34-4_amd64.deb ... 2022-03-02T06:46:36.990 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-03-02T06:46:37.422 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package xmlstarlet. 2022-03-02T06:46:37.435 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../081-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-03-02T06:46:37.481 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-03-02T06:46:37.985 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-test. 2022-03-02T06:46:37.999 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../082-ceph-test_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:38.044 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-test (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:43.300 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-volume. 2022-03-02T06:46:43.315 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../083-ceph-volume_17.0.0-10916-g8b0a50f4-1focal_all.deb ... 2022-03-02T06:46:43.395 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-volume (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:44.065 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephadm. 2022-03-02T06:46:44.082 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../084-cephadm_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:44.118 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephadm (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:44.495 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs-dev. 2022-03-02T06:46:44.505 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../085-libcephfs-dev_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:44.559 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs-dev (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:46.041 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package nvme-cli. 2022-03-02T06:46:46.055 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../086-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-03-02T06:46:46.181 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-03-02T06:46:46.648 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-atomicwrites. 2022-03-02T06:46:46.662 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../087-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-03-02T06:46:46.708 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-03-02T06:46:47.067 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-attr. 2022-03-02T06:46:47.077 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../088-python3-attr_19.3.0-2_all.deb ... 2022-03-02T06:46:47.116 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-03-02T06:46:47.610 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webencodings. 2022-03-02T06:46:47.619 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../089-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-03-02T06:46:47.670 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-02T06:46:48.078 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-html5lib. 2022-03-02T06:46:48.089 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../090-python3-html5lib_1.0.1-2_all.deb ... 2022-03-02T06:46:48.127 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-03-02T06:46:48.632 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-03-02T06:46:48.640 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../091-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-03-02T06:46:48.682 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-02T06:46:49.202 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-packaging. 2022-03-02T06:46:49.212 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../092-python3-packaging_20.3-1_all.deb ... 2022-03-02T06:46:49.262 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-packaging (20.3-1) ... 2022-03-02T06:46:49.871 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastescript. 2022-03-02T06:46:49.884 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../093-python3-pastescript_2.0.2-4_all.deb ... 2022-03-02T06:46:49.920 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-03-02T06:46:50.371 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pluggy. 2022-03-02T06:46:50.387 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../094-python3-pluggy_0.13.0-2_all.deb ... 2022-03-02T06:46:50.429 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-03-02T06:46:50.848 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-psutil. 2022-03-02T06:46:50.861 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../095-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-03-02T06:46:50.906 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-03-02T06:46:51.309 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-py. 2022-03-02T06:46:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../096-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-03-02T06:46:51.363 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-03-02T06:46:51.857 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyinotify. 2022-03-02T06:46:51.871 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../097-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-03-02T06:46:51.967 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-02T06:46:52.409 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-wcwidth. 2022-03-02T06:46:52.422 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../098-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-03-02T06:46:52.467 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-02T06:46:52.820 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pytest. 2022-03-02T06:46:52.833 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-03-02T06:46:52.913 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-03-02T06:46:53.409 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplejson. 2022-03-02T06:46:53.421 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-03-02T06:46:53.458 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-02T06:46:53.826 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package qttranslations5-l10n. 2022-03-02T06:46:53.839 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../101-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2022-03-02T06:46:53.920 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2022-03-02T06:46:55.040 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package radosgw. 2022-03-02T06:46:55.054 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../102-radosgw_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:55.134 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking radosgw (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:56.710 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package rbd-fuse. 2022-03-02T06:46:56.723 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../103-rbd-fuse_17.0.0-10916-g8b0a50f4-1focal_amd64.deb ... 2022-03-02T06:46:56.781 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking rbd-fuse (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:57.086 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-attr (19.3.0-2) ... 2022-03-02T06:46:57.472 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-03-02T06:46:57.685 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2022-03-02T06:46:57.816 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-03-02T06:46:58.063 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephadm (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:46:58.933 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user cephadm....done 2022-03-02T06:46:59.116 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-waitress (1.4.1-1) ... 2022-03-02T06:46:59.464 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-03-02T06:46:59.889 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-03-02T06:47:00.230 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-03-02T06:47:00.576 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-03-02T06:47:01.004 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-03-02T06:47:01.332 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-03-02T06:47:01.603 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-02T06:47:01.925 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-02T06:47:02.278 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-argparse (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:02.562 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-02T06:47:02.667 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-03-02T06:47:02.973 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-02T06:47:03.277 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libpcre2-16-0:amd64 (10.34-7) ... 2022-03-02T06:47:03.523 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-03-02T06:47:03.929 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-03-02T06:47:04.201 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-02T06:47:04.696 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-03-02T06:47:05.043 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-03-02T06:47:05.373 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-packaging (20.3-1) ... 2022-03-02T06:47:05.726 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-03-02T06:47:06.017 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2022-03-02T06:47:06.180 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-02T06:47:06.497 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-03-02T06:47:06.936 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-02T06:47:07.395 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-02T06:47:07.768 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-common (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:08.114 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-03-02T06:47:08.378 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-03-02T06:47:08.733 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-02T06:47:08.876 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-03-02T06:47:09.181 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-03-02T06:47:09.307 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-02T06:47:09.495 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-03-02T06:47:09.741 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-03-02T06:47:10.105 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-03-02T06:47:10.467 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-03-02T06:47:10.777 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-02T06:47:10.925 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-03-02T06:47:11.255 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-03-02T06:47:11.581 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-02T06:47:11.903 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-03-02T06:47:12.082 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-03-02T06:47:12.346 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-03-02T06:47:12.517 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-02T06:47:12.858 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-03-02T06:47:13.243 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-02T06:47:13.377 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-02T06:47:13.612 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-03-02T06:47:14.020 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-02T06:47:14.413 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-03-02T06:47:16.076 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-02T06:47:16.202 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-03-02T06:47:16.936 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-03-02T06:47:17.805 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-03-02T06:47:18.123 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-03-02T06:47:18.262 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-03-02T06:47:18.762 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-02T06:47:19.130 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-02T06:47:19.521 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-02T06:47:20.733 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-02T06:47:20.868 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-03-02T06:47:21.117 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-02T06:47:21.514 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-03-02T06:47:21.614 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rsa (4.0-3) ... 2022-03-02T06:47:21.938 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2022-03-02T06:47:22.114 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-03-02T06:47:22.467 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-modules-core (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:22.593 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-02T06:47:22.911 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-03-02T06:47:23.230 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-03-02T06:47:23.681 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librados2 (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:23.841 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librgw2 (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:24.036 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libsqlite3-mod-ceph (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:24.160 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs2 (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:24.328 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-routes (2.4.1-2) ... 2022-03-02T06:47:24.674 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libradosstriper1 (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:24.966 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-03-02T06:47:25.297 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librbd1 (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:25.473 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-fuse (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:25.777 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-02T06:47:25.777 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-02T06:47:26.699 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs-dev (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:26.851 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rados (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:26.977 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-03-02T06:47:28.064 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rbd (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:28.305 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up rbd-fuse (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:28.473 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rgw (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:28.616 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cephfs (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:28.860 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-common (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:29.515 INFO:teuthology.orchestra.run.smithi136.stdout:Adding group ceph....done 2022-03-02T06:47:30.064 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user ceph....done 2022-03-02T06:47:30.283 INFO:teuthology.orchestra.run.smithi136.stdout:Setting system user ceph properties....done 2022-03-02T06:47:30.303 INFO:teuthology.orchestra.run.smithi136.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-03-02T06:47:30.459 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-03-02T06:47:30.978 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-03-02T06:47:31.891 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-test (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:32.148 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up radosgw (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:32.443 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-02T06:47:32.443 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-02T06:47:34.110 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-base (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:34.715 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-03-02T06:47:35.614 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mds (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:35.863 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-02T06:47:35.863 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-02T06:47:36.784 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:37.048 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-02T06:47:37.048 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-02T06:47:37.886 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-osd (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:38.761 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-02T06:47:38.761 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-02T06:47:39.679 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-k8sevents (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:39.912 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:40.232 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mon (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:40.487 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-02T06:47:40.488 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-02T06:47:41.359 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:41.483 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-dashboard (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:41.651 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-volume (17.0.0-10916-g8b0a50f4-1focal) ... 2022-03-02T06:47:41.753 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for systemd (245.4-4ubuntu3.15) ... 2022-03-02T06:47:42.339 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-02T06:47:45.045 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-02T06:47:45.595 INFO:teuthology.orchestra.run.smithi136.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-03-02T06:47:45.598 DEBUG:teuthology.parallel:result is None 2022-03-02T06:47:45.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo lsb_release -is 2022-03-02T06:47:45.647 INFO:teuthology.orchestra.run.smithi136.stdout:Ubuntu 2022-03-02T06:47:45.653 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-02T06:47:45.654 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2 2022-03-02T06:47:45.871 DEBUG:teuthology.orchestra.run.smithi136:> dpkg-query -W -f '${Version}' ceph 2022-03-02T06:47:45.893 INFO:teuthology.orchestra.run.smithi136.stdout:17.0.0-10916-g8b0a50f4-1focal 2022-03-02T06:47:45.895 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-10916-g8b0a50f4-1focal 2022-03-02T06:47:45.896 INFO:teuthology.task.install:The correct ceph version 17.0.0-10916-g8b0a50f4-1focal is installed. 2022-03-02T06:47:45.897 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-02T06:47:45.897 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:47:45.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-02T06:47:45.956 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-02T06:47:45.956 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:47:45.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2022-03-02T06:47:45.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-02T06:47:46.027 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-02T06:47:46.027 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2022-03-02T06:47:46.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-02T06:47:46.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-02T06:47:46.050 INFO:teuthology.run_tasks:Running task radosbench... 2022-03-02T06:47:46.065 INFO:tasks.radosbench:Beginning radosbench... 2022-03-02T06:47:46.065 INFO:tasks.rook.ceph_manager.ceph:creating pool_name unique_pool_0 2022-03-02T06:47:46.066 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd pool create unique_pool_0 16 2022-03-02T06:47:47.671 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_0' created 2022-03-02T06:47:47.688 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2022-03-02T06:47:49.338 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rados' on pool 'unique_pool_0' 2022-03-02T06:47:50.370 DEBUG:teuthology.orchestra.run.smithi136:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.a -b 65536 --object-size 65536 -p unique_pool_0 bench 360 write' 2022-03-02T06:47:50.373 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-02T06:47:50.386 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch host label add '`hostname`' 'foo 2022-03-02T06:47:50.387 DEBUG:teuthology.orchestra.run.smithi136:> ceph' orch host ls '|' grep 'foo 2022-03-02T06:47:50.387 DEBUG:teuthology.orchestra.run.smithi136:> ceph' orch host label rm '`hostname`' 'foo 2022-03-02T06:47:50.387 DEBUG:teuthology.orchestra.run.smithi136:> ceph' orch host ls '|' grep -v 'foo 2022-03-02T06:47:50.388 DEBUG:teuthology.orchestra.run.smithi136:> ' 2022-03-02T06:47:50.517 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:hints = 1 2022-03-02T06:47:50.517 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 360 seconds or 0 objects 2022-03-02T06:47:50.518 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Object prefix: benchmark_data_smithi136_43115 2022-03-02T06:47:50.533 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:47:50.533 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 0 0 0 0 0 0 - 0 2022-03-02T06:47:50.713 INFO:teuthology.orchestra.run.smithi136.stdout:ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-03-02T06:47:50.741 INFO:teuthology.run_tasks:Running task exec... 2022-03-02T06:47:50.755 INFO:teuthology.task.exec:Executing custom commands... 2022-03-02T06:47:50.755 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi136.front.sepia.ceph.com 2022-03-02T06:47:50.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-02T06:47:50.757 DEBUG:teuthology.orchestra.run.smithi136:> toolbox() { 2022-03-02T06:47:50.757 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-02T06:47:50.757 DEBUG:teuthology.orchestra.run.smithi136:> } 2022-03-02T06:47:50.758 DEBUG:teuthology.orchestra.run.smithi136:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-02T06:47:50.758 DEBUG:teuthology.orchestra.run.smithi136:> toolbox ceph orch osd rm 0 --force 2022-03-02T06:47:50.758 DEBUG:teuthology.orchestra.run.smithi136:> removed_pv="" 2022-03-02T06:47:50.759 DEBUG:teuthology.orchestra.run.smithi136:> while [ "$removed_pv" = "" ] 2022-03-02T06:47:50.759 DEBUG:teuthology.orchestra.run.smithi136:> do 2022-03-02T06:47:50.759 DEBUG:teuthology.orchestra.run.smithi136:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-02T06:47:50.759 DEBUG:teuthology.orchestra.run.smithi136:> sleep 3s 2022-03-02T06:47:50.760 DEBUG:teuthology.orchestra.run.smithi136:> done 2022-03-02T06:47:50.760 DEBUG:teuthology.orchestra.run.smithi136:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-02T06:47:50.760 DEBUG:teuthology.orchestra.run.smithi136:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-02T06:47:50.761 DEBUG:teuthology.orchestra.run.smithi136:> toolbox ceph orch device zap $host $target_path --force 2022-03-02T06:47:50.761 DEBUG:teuthology.orchestra.run.smithi136:> zap_completion="0" 2022-03-02T06:47:50.761 DEBUG:teuthology.orchestra.run.smithi136:> while [ "$zap_completion" = "0" ] 2022-03-02T06:47:50.761 DEBUG:teuthology.orchestra.run.smithi136:> do 2022-03-02T06:47:50.762 DEBUG:teuthology.orchestra.run.smithi136:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-02T06:47:50.762 DEBUG:teuthology.orchestra.run.smithi136:> sleep 3s 2022-03-02T06:47:50.762 DEBUG:teuthology.orchestra.run.smithi136:> done 2022-03-02T06:47:50.763 DEBUG:teuthology.orchestra.run.smithi136:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-02T06:47:50.763 DEBUG:teuthology.orchestra.run.smithi136:> toolbox ceph orch apply osd --all-available-devices 2022-03-02T06:47:50.763 DEBUG:teuthology.orchestra.run.smithi136:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-02T06:47:50.764 DEBUG:teuthology.orchestra.run.smithi136:> num_osd="0" 2022-03-02T06:47:50.764 DEBUG:teuthology.orchestra.run.smithi136:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-02T06:47:50.764 DEBUG:teuthology.orchestra.run.smithi136:> do 2022-03-02T06:47:50.765 DEBUG:teuthology.orchestra.run.smithi136:> echo "waiting for osd to come back up" 2022-03-02T06:47:50.765 DEBUG:teuthology.orchestra.run.smithi136:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-02T06:47:50.765 DEBUG:teuthology.orchestra.run.smithi136:> sleep 30s 2022-03-02T06:47:50.766 DEBUG:teuthology.orchestra.run.smithi136:> done 2022-03-02T06:47:50.766 DEBUG:teuthology.orchestra.run.smithi136:> ' 2022-03-02T06:47:50.779 INFO:teuthology.orchestra.run.smithi136.stderr:++ toolbox ceph osd stat 2022-03-02T06:47:50.779 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f3 -d ' ' 2022-03-02T06:47:50.779 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-02T06:47:51.533 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 1 16 1395 1379 86.14 86.1875 0.0107788 0.011526 2022-03-02T06:47:52.535 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 2 16 2734 2718 84.8985 83.6875 0.00919683 0.0117481 2022-03-02T06:47:53.162 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:47:53.535 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 3 16 4106 4090 85.1794 85.75 0.0169006 0.0117117 2022-03-02T06:47:53.935 INFO:teuthology.orchestra.run.smithi136.stderr:+ orig_num_osd=4 2022-03-02T06:47:53.935 INFO:teuthology.orchestra.run.smithi136.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-02T06:47:53.935 INFO:teuthology.orchestra.run.smithi136.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-02T06:47:54.016 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:47:54.534 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 4 16 5267 5251 82.024 72.5625 0.0152697 0.0121538 2022-03-02T06:47:54.915 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:47:54.911886 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-02T06:47:54.915 INFO:tasks.rook.operator.smithi136.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", AllowUnsupported: true}, 2022-03-02T06:47:54.915 INFO:tasks.rook.operator.smithi136.stdout: Storage: v1.StorageScopeSpec{ 2022-03-02T06:47:54.916 INFO:tasks.rook.operator.smithi136.stdout: ... // 3 identical fields 2022-03-02T06:47:54.916 INFO:tasks.rook.operator.smithi136.stdout: Config: nil, 2022-03-02T06:47:54.916 INFO:tasks.rook.operator.smithi136.stdout: Selection: {}, 2022-03-02T06:47:54.916 INFO:tasks.rook.operator.smithi136.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-02T06:47:54.916 INFO:tasks.rook.operator.smithi136.stdout: {Name: "smithi136-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi136-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-02T06:47:54.917 INFO:tasks.rook.operator.smithi136.stdout:- { 2022-03-02T06:47:54.917 INFO:tasks.rook.operator.smithi136.stdout:- Name: "smithi136-lv-2", 2022-03-02T06:47:54.917 INFO:tasks.rook.operator.smithi136.stdout:- Count: 1, 2022-03-02T06:47:54.917 INFO:tasks.rook.operator.smithi136.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-02T06:47:54.918 INFO:tasks.rook.operator.smithi136.stdout:- }, 2022-03-02T06:47:54.918 INFO:tasks.rook.operator.smithi136.stdout: {Name: "smithi136-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi136-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-02T06:47:54.918 INFO:tasks.rook.operator.smithi136.stdout: {Name: "smithi136-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi136-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-02T06:47:54.919 INFO:tasks.rook.operator.smithi136.stdout: }, 2022-03-02T06:47:54.919 INFO:tasks.rook.operator.smithi136.stdout: }, 2022-03-02T06:47:54.919 INFO:tasks.rook.operator.smithi136.stdout: Annotations: nil, 2022-03-02T06:47:54.919 INFO:tasks.rook.operator.smithi136.stdout: Labels: nil, 2022-03-02T06:47:54.919 INFO:tasks.rook.operator.smithi136.stdout: ... // 20 identical fields 2022-03-02T06:47:54.920 INFO:tasks.rook.operator.smithi136.stdout: } 2022-03-02T06:47:54.920 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:47:54.911957 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:47:55.016 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:47:55.015409 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:47:55.072 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:47:55.071761 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2... 2022-03-02T06:47:55.534 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 5 16 6289 6273 78.3927 63.875 0.00711149 0.0123969 2022-03-02T06:47:56.535 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 6 16 6289 6273 65.3288 0 - 0.0123969 2022-03-02T06:47:57.535 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 7 16 6289 6273 55.9969 0 - 0.0123969 2022-03-02T06:47:58.535 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 8 16 6289 6273 48.9976 0 - 0.0123969 2022-03-02T06:47:59.535 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 9 16 6289 6273 43.5537 0 - 0.0123969 2022-03-02T06:48:00.535 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 10 16 6289 6273 39.1987 0 - 0.0123969 2022-03-02T06:48:01.535 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 11 16 6289 6273 35.6354 0 - 0.0123969 2022-03-02T06:48:02.536 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 12 16 6289 6273 32.666 0 - 0.0123969 2022-03-02T06:48:02.937 INFO:teuthology.orchestra.run.smithi136.stdout:Success 2022-03-02T06:48:02.977 INFO:teuthology.orchestra.run.smithi136.stderr:+ removed_pv= 2022-03-02T06:48:02.977 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' '' = '' ']' 2022-03-02T06:48:02.977 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl get pv 2022-03-02T06:48:02.978 INFO:teuthology.orchestra.run.smithi136.stderr:++ grep Released 2022-03-02T06:48:02.978 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f1 -d ' ' 2022-03-02T06:48:03.034 INFO:teuthology.orchestra.run.smithi136.stderr:+ removed_pv= 2022-03-02T06:48:03.034 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 3s 2022-03-02T06:48:03.536 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 13 16 6289 6273 30.1533 0 - 0.0123969 2022-03-02T06:48:04.536 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 14 16 6289 6273 27.9997 0 - 0.0123969 2022-03-02T06:48:05.536 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 15 16 6462 6446 26.8539 1.08125 0.0127137 0.0372106 2022-03-02T06:48:06.035 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' '' = '' ']' 2022-03-02T06:48:06.035 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl get pv 2022-03-02T06:48:06.036 INFO:teuthology.orchestra.run.smithi136.stderr:++ grep Released 2022-03-02T06:48:06.037 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f1 -d ' ' 2022-03-02T06:48:06.209 INFO:teuthology.orchestra.run.smithi136.stderr:+ removed_pv= 2022-03-02T06:48:06.209 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 3s 2022-03-02T06:48:06.537 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 16 16 7375 7359 28.7411 57.0625 0.00719027 0.0347591 2022-03-02T06:48:07.537 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 17 16 8333 8317 30.572 59.875 0.0193881 0.0326794 2022-03-02T06:48:08.537 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 18 16 9370 9354 32.4738 64.8125 0.0210193 0.0307627 2022-03-02T06:48:08.544 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:08.542926 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-02T06:48:09.210 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' '' = '' ']' 2022-03-02T06:48:09.211 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl get pv 2022-03-02T06:48:09.211 INFO:teuthology.orchestra.run.smithi136.stderr:++ grep Released 2022-03-02T06:48:09.211 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f1 -d ' ' 2022-03-02T06:48:09.277 INFO:teuthology.orchestra.run.smithi136.stderr:+ removed_pv=smithi136-lv-2 2022-03-02T06:48:09.277 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 3s 2022-03-02T06:48:09.537 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 19 16 10330 10314 33.9221 60 0.0115569 0.0294607 2022-03-02T06:48:10.537 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:48:10.536561+0000 min lat: 0.00139821 max lat: 9.97622 avg lat: 0.0286991 2022-03-02T06:48:10.538 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:48:10.538 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 20 16 11162 11146 34.8257 52 0.0161806 0.0286991 2022-03-02T06:48:11.537 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 21 16 11885 11869 35.3189 45.1875 0.0152984 0.0282979 2022-03-02T06:48:12.278 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' smithi136-lv-2 = '' ']' 2022-03-02T06:48:12.278 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl get pv smithi136-lv-2 -o 'jsonpath={.spec.local.path}' 2022-03-02T06:48:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:+ target_path=/dev/vg_nvme/lv_2 2022-03-02T06:48:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:++ echo smithi136-lv-2 2022-03-02T06:48:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f1 -d - 2022-03-02T06:48:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:+ host=smithi136 2022-03-02T06:48:12.422 INFO:teuthology.orchestra.run.smithi136.stderr:+ toolbox ceph orch device zap smithi136 /dev/vg_nvme/lv_2 --force 2022-03-02T06:48:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi136 /dev/vg_nvme/lv_2 --force 2022-03-02T06:48:12.538 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 22 16 12637 12621 35.8491 47 0.0269134 0.0278748 2022-03-02T06:48:12.772 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:48:13.538 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 23 16 13479 13463 36.5782 52.625 0.0232511 0.0273156 2022-03-02T06:48:13.550 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/vg_nvme/lv_2 on smithi136 zapped 2022-03-02T06:48:13.585 INFO:teuthology.orchestra.run.smithi136.stderr:+ zap_completion=0 2022-03-02T06:48:13.585 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' 0 = 0 ']' 2022-03-02T06:48:13.585 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-02T06:48:13.672 INFO:teuthology.orchestra.run.smithi136.stderr:+ zap_completion= 2022-03-02T06:48:13.672 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 3s 2022-03-02T06:48:14.538 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 24 16 14176 14160 36.869 43.5625 0.0192889 0.0271066 2022-03-02T06:48:15.539 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 25 16 14930 14914 37.278 47.125 0.0164422 0.0268108 2022-03-02T06:48:16.541 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 26 16 15573 15557 37.3897 40.1875 0.0114335 0.026735 2022-03-02T06:48:16.673 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' '' = 0 ']' 2022-03-02T06:48:16.673 INFO:teuthology.orchestra.run.smithi136.stderr:+ kubectl patch pv smithi136-lv-2 -p '{"spec":{"claimRef": null}}' 2022-03-02T06:48:16.747 INFO:teuthology.orchestra.run.smithi136.stdout:persistentvolume/smithi136-lv-2 patched 2022-03-02T06:48:16.752 INFO:teuthology.orchestra.run.smithi136.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-02T06:48:16.752 INFO:teuthology.orchestra.run.smithi136.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-02T06:48:17.539 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 27 16 16274 16258 37.6274 43.8125 0.0666472 0.0265378 2022-03-02T06:48:17.794 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:48:18.539 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 28 16 16995 16979 37.8928 45.0625 0.0126541 0.0263554 2022-03-02T06:48:19.141 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:19.138272 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-02T06:48:19.141 INFO:tasks.rook.operator.smithi136.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", AllowUnsupported: true}, 2022-03-02T06:48:19.149 INFO:tasks.rook.operator.smithi136.stdout: Storage: v1.StorageScopeSpec{ 2022-03-02T06:48:19.149 INFO:tasks.rook.operator.smithi136.stdout: ... // 3 identical fields 2022-03-02T06:48:19.149 INFO:tasks.rook.operator.smithi136.stdout: Config: nil, 2022-03-02T06:48:19.149 INFO:tasks.rook.operator.smithi136.stdout: Selection: {}, 2022-03-02T06:48:19.150 INFO:tasks.rook.operator.smithi136.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-02T06:48:19.150 INFO:tasks.rook.operator.smithi136.stdout: {Name: "smithi136-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi136-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-02T06:48:19.150 INFO:tasks.rook.operator.smithi136.stdout: {Name: "smithi136-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi136-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-02T06:48:19.150 INFO:tasks.rook.operator.smithi136.stdout: {Name: "smithi136-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi136-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-02T06:48:19.151 INFO:tasks.rook.operator.smithi136.stdout:+ { 2022-03-02T06:48:19.151 INFO:tasks.rook.operator.smithi136.stdout:+ Name: "smithi136-lv-2", 2022-03-02T06:48:19.151 INFO:tasks.rook.operator.smithi136.stdout:+ Count: 1, 2022-03-02T06:48:19.152 INFO:tasks.rook.operator.smithi136.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-02T06:48:19.152 INFO:tasks.rook.operator.smithi136.stdout:+ }, 2022-03-02T06:48:19.152 INFO:tasks.rook.operator.smithi136.stdout: }, 2022-03-02T06:48:19.152 INFO:tasks.rook.operator.smithi136.stdout: }, 2022-03-02T06:48:19.153 INFO:tasks.rook.operator.smithi136.stdout: Annotations: nil, 2022-03-02T06:48:19.153 INFO:tasks.rook.operator.smithi136.stdout: Labels: nil, 2022-03-02T06:48:19.153 INFO:tasks.rook.operator.smithi136.stdout: ... // 20 identical fields 2022-03-02T06:48:19.153 INFO:tasks.rook.operator.smithi136.stdout: } 2022-03-02T06:48:19.539 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 29 16 17720 17704 38.1485 45.3125 0.0184812 0.0261983 2022-03-02T06:48:20.539 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 30 16 18421 18405 38.3371 43.8125 0.194022 0.0260218 2022-03-02T06:48:21.014 INFO:teuthology.orchestra.run.smithi136.stdout:['No change'] 2022-03-02T06:48:21.047 INFO:teuthology.orchestra.run.smithi136.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-02T06:48:21.540 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 31 16 19112 19096 38.4934 43.1875 0.0398209 0.0259475 2022-03-02T06:48:21.920 INFO:teuthology.orchestra.run.smithi136.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-02T06:48:22.540 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 32 16 19795 19779 38.6243 42.6875 0.0306728 0.0258747 2022-03-02T06:48:22.630 INFO:teuthology.orchestra.run.smithi136.stderr:+ num_osd=0 2022-03-02T06:48:22.631 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' 0 '!=' 4 ']' 2022-03-02T06:48:22.631 INFO:teuthology.orchestra.run.smithi136.stderr:+ echo 'waiting for osd to come back up' 2022-03-02T06:48:22.631 INFO:teuthology.orchestra.run.smithi136.stdout:waiting for osd to come back up 2022-03-02T06:48:22.632 INFO:teuthology.orchestra.run.smithi136.stderr:++ toolbox ceph osd stat 2022-03-02T06:48:22.632 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-02T06:48:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f3 -d ' ' 2022-03-02T06:48:23.540 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 33 16 20392 20376 38.5845 37.3125 0.047048 0.0258968 2022-03-02T06:48:24.542 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 34 16 21106 21090 38.762 44.625 0.0138903 0.025789 2022-03-02T06:48:24.594 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:48:25.453 INFO:teuthology.orchestra.run.smithi136.stderr:+ num_osd=3 2022-03-02T06:48:25.454 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 30s 2022-03-02T06:48:25.540 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 35 16 21676 21660 38.6722 35.625 0.0106034 0.0257849 2022-03-02T06:48:26.540 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 36 16 22168 22152 38.4521 30.75 0.0124321 0.0259579 2022-03-02T06:48:27.508 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:27.504562 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10916 quincy" 2022-03-02T06:48:27.508 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:27.504590 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-02T06:48:27.509 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:27.504612 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10916 quincy", pursuing 2022-03-02T06:48:27.540 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 37 16 22941 22925 38.7184 48.3125 0.0183399 0.0258142 2022-03-02T06:48:28.092 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:28.060071 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:48:28.541 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 38 16 23390 23374 38.438 28.0625 0.0213256 0.0260033 2022-03-02T06:48:29.161 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:29.159104 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:48:29.210 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:29.206243 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:48:29.541 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 39 16 24201 24185 38.7519 50.6875 0.018695 0.0257922 2022-03-02T06:48:30.541 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:48:30.539620+0000 min lat: 0.00139821 max lat: 9.97622 avg lat: 0.0257699 2022-03-02T06:48:30.541 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:48:30.542 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 40 16 24766 24750 38.6658 35.3125 0.213836 0.0257699 2022-03-02T06:48:30.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:30.762743 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10916 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2" 2022-03-02T06:48:31.541 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 41 16 25606 25590 39.0031 52.5 0.0339573 0.025627 2022-03-02T06:48:31.681 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:31.675224 I | op-mon: start running mons 2022-03-02T06:48:31.715 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:31.712380 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:48:31.754 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:31.751893 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:48:31.775 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:31.768903 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:48:31.775 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:31.770667 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:48:32.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:32.485437 I | op-mon: targeting the mon count 1 2022-03-02T06:48:32.528 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:32.524096 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-02T06:48:32.536 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:32.531957 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:48:32.541 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 42 16 26180 26164 38.9286 35.875 0.0214978 0.0256678 2022-03-02T06:48:32.959 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:32.954075 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:48:32.985 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:32.983154 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2... 2022-03-02T06:48:33.543 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 43 16 26760 26744 38.8662 36.25 0.0262223 0.0257157 2022-03-02T06:48:33.680 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:33.670525 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-02T06:48:34.218 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:34.213327 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-02T06:48:34.542 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 44 16 27414 27398 38.9118 40.875 0.00972888 0.0256844 2022-03-02T06:48:35.542 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 45 16 28082 28066 38.9747 41.75 0.0176979 0.0256472 2022-03-02T06:48:36.402 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:36.399055 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-02T06:48:36.543 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 46 16 28467 28451 38.6505 24.0625 0.014134 0.025807 2022-03-02T06:48:37.543 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 47 16 28857 28841 38.3468 24.375 0.0218002 0.026063 2022-03-02T06:48:38.543 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 48 16 29475 29459 38.3525 38.625 0.0346512 0.02599 2022-03-02T06:48:38.601 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:38.597880 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-02T06:48:39.543 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 49 16 29475 29459 37.5698 0 - 0.02599 2022-03-02T06:48:40.543 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 50 16 29875 29859 37.3184 12.5 0.0129421 0.0267875 2022-03-02T06:48:41.075 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:41.071736 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-02T06:48:41.544 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 51 16 30593 30577 37.4665 44.875 0.0104767 0.0266809 2022-03-02T06:48:42.544 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 52 16 31228 31212 37.5091 39.6875 0.0141776 0.0266482 2022-03-02T06:48:43.336 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:43.331407 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-02T06:48:43.544 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 53 16 31822 31806 37.5018 37.125 0.00533782 0.02654 2022-03-02T06:48:44.544 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 54 16 31916 31900 36.9161 5.875 0.00342814 0.0264828 2022-03-02T06:48:45.341 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:45.337411 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-02T06:48:45.544 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 55 16 31916 31900 36.2449 0 - 0.0264828 2022-03-02T06:48:46.545 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 56 16 31916 31900 35.5977 0 - 0.0264828 2022-03-02T06:48:47.344 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:48:47.340309 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-02T06:48:47.545 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 57 16 31916 31900 34.9732 0 - 0.0264828 2022-03-02T06:48:48.545 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 58 16 31916 31900 34.3703 0 - 0.0264828 2022-03-02T06:48:49.545 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 59 16 31916 31900 33.7877 0 - 0.0264828 2022-03-02T06:48:50.548 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:48:50.541574+0000 min lat: 0.00139821 max lat: 9.97622 avg lat: 0.0264828 2022-03-02T06:48:50.549 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:48:50.549 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 60 16 31916 31900 33.2246 0 - 0.0264828 2022-03-02T06:48:51.545 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 61 16 31916 31900 32.68 0 - 0.0264828 2022-03-02T06:48:52.545 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 62 16 32113 32097 32.3515 1.53906 0.00345564 0.0307097 2022-03-02T06:48:53.546 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 63 16 32113 32097 31.838 0 - 0.0307097 2022-03-02T06:48:54.546 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 64 16 32113 32097 31.3405 0 - 0.0307097 2022-03-02T06:48:55.457 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' 3 '!=' 4 ']' 2022-03-02T06:48:55.458 INFO:teuthology.orchestra.run.smithi136.stderr:+ echo 'waiting for osd to come back up' 2022-03-02T06:48:55.459 INFO:teuthology.orchestra.run.smithi136.stdout:waiting for osd to come back up 2022-03-02T06:48:55.460 INFO:teuthology.orchestra.run.smithi136.stderr:++ toolbox ceph osd stat 2022-03-02T06:48:55.460 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-02T06:48:55.461 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f3 -d ' ' 2022-03-02T06:48:55.546 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 65 16 32113 32097 30.8584 0 - 0.0307097 2022-03-02T06:48:56.546 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 66 16 32113 32097 30.3908 0 - 0.0307097 2022-03-02T06:48:57.144 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:48:57.547 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 67 16 32113 32097 29.9372 0 - 0.0307097 2022-03-02T06:48:58.115 INFO:teuthology.orchestra.run.smithi136.stderr:+ num_osd=3 2022-03-02T06:48:58.115 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 30s 2022-03-02T06:48:58.547 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 68 16 32113 32097 29.497 0 - 0.0307097 2022-03-02T06:48:59.547 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 69 16 32113 32097 29.0695 0 - 0.0307097 2022-03-02T06:49:00.547 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 70 16 32497 32481 28.997 3 0.0174238 0.0344773 2022-03-02T06:49:01.547 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 71 16 33123 33107 29.1396 39.125 0.0117353 0.0343066 2022-03-02T06:49:02.549 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 72 16 33688 33672 29.2252 35.3125 0.0301588 0.0341174 2022-03-02T06:49:03.548 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 73 16 33997 33981 29.0894 19.3125 0.025775 0.0343505 2022-03-02T06:49:04.548 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 74 16 34555 34539 29.1676 34.875 0.0168885 0.0342685 2022-03-02T06:49:05.548 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 75 16 35261 35245 29.3669 44.125 0.0495221 0.034042 2022-03-02T06:49:06.548 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 76 16 35787 35771 29.413 32.875 0.104625 0.033977 2022-03-02T06:49:07.548 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 77 16 36408 36392 29.5351 38.8125 0.0307256 0.0338478 2022-03-02T06:49:08.549 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 78 16 36984 36968 29.6177 36 0.0174881 0.0337543 2022-03-02T06:49:09.549 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 79 15 37604 37589 29.7341 38.8125 0.0158728 0.0336215 2022-03-02T06:49:10.551 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:49:10.544272+0000 min lat: 0.00139821 max lat: 9.97622 avg lat: 0.0334951 2022-03-02T06:49:10.551 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:49:10.552 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 80 16 38227 38211 29.8483 38.875 0.0160735 0.0334951 2022-03-02T06:49:11.549 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 81 16 38766 38750 29.8955 33.6875 0.0136357 0.0334417 2022-03-02T06:49:12.550 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 82 16 39229 39213 29.8838 28.9375 0.0148997 0.0334521 2022-03-02T06:49:13.550 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 83 16 39800 39784 29.9537 35.6875 0.0157786 0.0333464 2022-03-02T06:49:14.550 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 84 16 40476 40460 30.1 42.25 0.0250846 0.0332111 2022-03-02T06:49:15.550 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 85 16 41102 41086 30.2061 39.125 0.0533303 0.0330925 2022-03-02T06:49:16.550 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 86 16 41782 41766 30.349 42.5 0.0224917 0.0329386 2022-03-02T06:49:17.554 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 87 16 42345 42329 30.4033 35.1875 0.0130716 0.0328803 2022-03-02T06:49:18.554 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 88 16 42796 42780 30.3781 28.1875 0.0195113 0.0329112 2022-03-02T06:49:19.554 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 89 16 43362 43346 30.4342 35.375 0.0160648 0.0328477 2022-03-02T06:49:20.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 90 16 43937 43921 30.4953 35.9375 0.0252312 0.0327831 2022-03-02T06:49:21.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 91 16 44467 44451 30.5242 33.125 0.00469552 0.032638 2022-03-02T06:49:22.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 92 16 44473 44457 30.1965 0.375 0.00334901 0.0326349 2022-03-02T06:49:23.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 93 16 44473 44457 29.8718 0 - 0.0326349 2022-03-02T06:49:24.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 94 16 44473 44457 29.554 0 - 0.0326349 2022-03-02T06:49:25.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 95 16 44473 44457 29.243 0 - 0.0326349 2022-03-02T06:49:26.065 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:26.059603 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10916 quincy" 2022-03-02T06:49:26.066 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:26.059619 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-02T06:49:26.066 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:26.059627 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10916 quincy", pursuing 2022-03-02T06:49:26.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 96 16 44473 44457 28.9384 0 - 0.0326349 2022-03-02T06:49:26.639 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:26.633621 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:49:27.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 97 16 44473 44457 28.6401 0 - 0.0326349 2022-03-02T06:49:27.799 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:27.793025 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:49:27.799 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:27.793233 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:49:28.117 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' 3 '!=' 4 ']' 2022-03-02T06:49:28.118 INFO:teuthology.orchestra.run.smithi136.stdout:waiting for osd to come back up 2022-03-02T06:49:28.118 INFO:teuthology.orchestra.run.smithi136.stderr:+ echo 'waiting for osd to come back up' 2022-03-02T06:49:28.119 INFO:teuthology.orchestra.run.smithi136.stderr:++ toolbox ceph osd stat 2022-03-02T06:49:28.119 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-02T06:49:28.119 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f3 -d ' ' 2022-03-02T06:49:28.185 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:49:28.388 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:28.382178 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10916 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2" 2022-03-02T06:49:28.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 98 16 44473 44457 28.3478 0 - 0.0326349 2022-03-02T06:49:28.832 INFO:teuthology.orchestra.run.smithi136.stderr:+ num_osd=3 2022-03-02T06:49:28.832 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 30s 2022-03-02T06:49:28.943 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:28.937506 I | op-mon: start running mons 2022-03-02T06:49:29.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 99 16 44473 44457 28.0615 0 - 0.0326349 2022-03-02T06:49:29.587 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:29.580927 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:49:29.887 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:29.881504 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:49:29.988 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:29.982599 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:49:29.989 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:29.982732 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:49:30.190 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:30.183558 I | op-mon: targeting the mon count 1 2022-03-02T06:49:30.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:30.188396 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:49:30.535 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:30.528867 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:49:30.535 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:30.528898 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:49:30.555 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:49:30.550305+0000 min lat: 0.00139821 max lat: 9.97622 avg lat: 0.0326349 2022-03-02T06:49:30.556 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:49:30.556 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 100 16 44473 44457 27.7809 0 - 0.0326349 2022-03-02T06:49:30.873 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:30.866891 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:49:30.873 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:30.866916 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:49:31.244 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:31.237992 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:49:31.245 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:31.238024 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:49:31.556 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 101 16 44473 44457 27.5059 0 - 0.0326349 2022-03-02T06:49:31.577 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:31.570391 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:49:31.577 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:31.570432 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:49:31.920 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:31.914271 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:49:31.920 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:31.914290 I | op-config: deleting "log file" option from the mon configuration database 2022-03-02T06:49:32.317 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.310657 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-02T06:49:32.318 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.310691 I | op-mon: checking for basic quorum with existing mons 2022-03-02T06:49:32.471 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.464752 I | op-mon: mon "a" endpoint is [v2:10.103.147.119:3300,v1:10.103.147.119:6789] 2022-03-02T06:49:32.489 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.483123 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:49:32.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.488484 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:49:32.497 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.488686 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:49:32.556 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 102 16 44510 44494 27.2589 0.23125 0.00923225 0.0365195 2022-03-02T06:49:32.732 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.725963 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-02T06:49:32.741 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.734683 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-02T06:49:32.742 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.734714 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:49:32.924 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:32.917739 I | op-mon: mons running: [a] 2022-03-02T06:49:33.345 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:33.339098 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:49:33.346 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:33.339128 I | op-mon: mons created: 1 2022-03-02T06:49:33.556 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 103 16 44510 44494 26.9943 0 - 0.0365195 2022-03-02T06:49:33.758 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:33.752376 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:49:33.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:33.759254 I | op-mon: mons running: [a] 2022-03-02T06:49:34.207 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:34.200891 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:49:34.207 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:34.200937 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-02T06:49:34.556 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 104 16 44510 44494 26.7347 0 - 0.0365195 2022-03-02T06:49:34.620 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:34.614780 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-02T06:49:35.068 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:35.062618 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-02T06:49:35.473 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:35.467140 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-02T06:49:35.556 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 105 16 44510 44494 26.4801 0 - 0.0365195 2022-03-02T06:49:35.966 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:35.959993 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-02T06:49:35.967 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:35.960012 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-02T06:49:36.403 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:36.396615 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-02T06:49:36.556 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 106 16 44510 44494 26.2303 0 - 0.0365195 2022-03-02T06:49:36.815 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:36.809071 I | cephclient: successfully enabled msgr2 protocol 2022-03-02T06:49:36.815 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:36.809108 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-02T06:49:36.816 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:36.809116 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-02T06:49:37.270 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:37.264122 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-02T06:49:37.295 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:37.289388 I | op-mgr: start running mgr 2022-03-02T06:49:37.295 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:37.289418 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-02T06:49:37.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 107 16 44510 44494 25.9852 0 - 0.0365195 2022-03-02T06:49:37.773 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:37.767822 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-02T06:49:37.781 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:37.775477 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-02T06:49:37.783 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:37.777634 I | op-mgr: setting services to point to mgr "a" 2022-03-02T06:49:37.998 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:37.991307 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-02T06:49:38.000 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:37.991339 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-02T06:49:38.020 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.013992 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-02T06:49:38.029 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.023207 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi136-lv-1" 2022-03-02T06:49:38.030 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.023247 I | op-osd: OSD PVC "smithi136-lv-1-data-04lnm9" already exists 2022-03-02T06:49:38.030 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.023258 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi136-lv-3" 2022-03-02T06:49:38.030 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.023268 I | op-osd: OSD PVC "smithi136-lv-3-data-0c469g" already exists 2022-03-02T06:49:38.030 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.023275 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi136-lv-4" 2022-03-02T06:49:38.031 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.023284 I | op-osd: OSD PVC "smithi136-lv-4-data-07m48t" already exists 2022-03-02T06:49:38.031 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.023292 I | op-osd: creating 1 new PVCs for device set "smithi136-lv-2" 2022-03-02T06:49:38.123 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.116759 I | op-osd: successfully provisioned PVC "smithi136-lv-2-data-0mp4gf" 2022-03-02T06:49:38.148 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.138042 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-1-data-04lnm9" 2022-03-02T06:49:38.149 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.138068 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-3-data-0c469g" 2022-03-02T06:49:38.149 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.138074 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-4-data-07m48t" 2022-03-02T06:49:38.149 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.138078 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-2-data-0mp4gf" 2022-03-02T06:49:38.556 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 108 16 44510 44494 25.7446 0 - 0.0365195 2022-03-02T06:49:38.573 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.567708 I | op-osd: started OSD provisioning job for PVC "smithi136-lv-2-data-0mp4gf" 2022-03-02T06:49:38.574 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.567739 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-02T06:49:38.574 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.567750 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-02T06:49:38.719 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.713019 I | op-osd: OSD orchestration status for PVC smithi136-lv-2-data-0mp4gf is "starting" 2022-03-02T06:49:38.760 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.754145 I | op-mgr: successful modules: dashboard 2022-03-02T06:49:38.760 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.754171 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-02T06:49:38.762 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:38.755035 I | op-mgr: successful modules: prometheus 2022-03-02T06:49:39.276 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:39.270234 I | op-mgr: successful modules: balancer 2022-03-02T06:49:39.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 109 16 44510 44494 25.5084 0 - 0.0365195 2022-03-02T06:49:39.896 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:39.890459 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-02T06:49:39.902 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:39.896840 I | op-osd: updating OSD 1 on PVC "smithi136-lv-1-data-04lnm9" 2022-03-02T06:49:39.903 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:39.896885 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-1-data-04lnm9" 2022-03-02T06:49:40.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 110 16 44778 44762 25.4288 2.09375 0.0138115 0.0393131 2022-03-02T06:49:40.687 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:40.681714 I | op-mgr: successful modules: orchestrator modules 2022-03-02T06:49:41.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 111 16 45383 45367 25.5403 37.8125 0.0404875 0.0391453 2022-03-02T06:49:41.871 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:41.860596 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-02T06:49:41.876 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:41.869398 I | op-osd: updating OSD 2 on PVC "smithi136-lv-3-data-0c469g" 2022-03-02T06:49:41.877 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:41.869422 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-3-data-0c469g" 2022-03-02T06:49:42.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 112 16 45942 45926 25.6242 34.9375 0.0388972 0.0390127 2022-03-02T06:49:43.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 113 16 46498 46482 25.7049 34.75 0.0430066 0.0388925 2022-03-02T06:49:44.009 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:44.002971 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-02T06:49:44.016 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:44.010374 I | op-osd: updating OSD 3 on PVC "smithi136-lv-4-data-07m48t" 2022-03-02T06:49:44.020 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:49:44.010398 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-4-data-07m48t" 2022-03-02T06:49:44.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 114 16 47126 47110 25.8237 39.25 0.0209894 0.0387147 2022-03-02T06:49:45.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 115 16 47835 47819 25.9844 44.3125 0.00892121 0.0384751 2022-03-02T06:49:46.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 116 16 48536 48520 26.138 43.8125 0.0346696 0.0382478 2022-03-02T06:49:47.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 117 16 48973 48957 26.1481 27.3125 0.0337938 0.0382351 2022-03-02T06:49:48.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 118 16 49629 49613 26.2739 41 0.0232853 0.0380532 2022-03-02T06:49:49.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 119 16 50035 50019 26.2663 25.375 0.023728 0.0380621 2022-03-02T06:49:50.557 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:49:50.552424+0000 min lat: 0.00139821 max lat: 11.1076 avg lat: 0.037902 2022-03-02T06:49:50.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:49:50.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 120 16 50669 50653 26.3776 39.625 0.0184415 0.037902 2022-03-02T06:49:51.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 121 16 50957 50941 26.3083 18 0.019127 0.0380009 2022-03-02T06:49:52.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 122 16 51494 51478 26.3678 33.5625 0.041771 0.0379148 2022-03-02T06:49:53.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 123 16 52006 51990 26.4135 32 0.0250815 0.0378514 2022-03-02T06:49:54.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 124 16 52469 52453 26.4339 28.9375 0.0329977 0.0378189 2022-03-02T06:49:55.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 125 16 52922 52906 26.4489 28.3125 0.0151847 0.0378006 2022-03-02T06:49:56.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 126 16 53531 53515 26.541 38.0625 0.0194036 0.0376701 2022-03-02T06:49:57.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 127 16 54051 54035 26.5879 32.5 0.0156772 0.0375997 2022-03-02T06:49:58.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 128 16 54640 54624 26.6677 36.8125 0.0174719 0.0374884 2022-03-02T06:49:58.832 INFO:teuthology.orchestra.run.smithi136.stdout:waiting for osd to come back up 2022-03-02T06:49:58.833 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' 3 '!=' 4 ']' 2022-03-02T06:49:58.833 INFO:teuthology.orchestra.run.smithi136.stderr:+ echo 'waiting for osd to come back up' 2022-03-02T06:49:58.834 INFO:teuthology.orchestra.run.smithi136.stderr:++ toolbox ceph osd stat 2022-03-02T06:49:58.834 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-02T06:49:58.834 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f3 -d ' ' 2022-03-02T06:49:59.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 129 16 55054 55038 26.6616 25.875 0.0199413 0.037487 2022-03-02T06:49:59.880 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:50:00.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 130 16 55557 55541 26.6983 31.4375 0.0255487 0.0374449 2022-03-02T06:50:01.136 INFO:teuthology.orchestra.run.smithi136.stderr:+ num_osd=3 2022-03-02T06:50:01.136 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 30s 2022-03-02T06:50:01.558 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 131 16 56143 56127 26.774 36.625 0.0386049 0.0373397 2022-03-02T06:50:02.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 132 16 56675 56659 26.8231 33.25 0.0187531 0.0372719 2022-03-02T06:50:03.562 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 133 16 57170 57154 26.854 30.9375 0.0246178 0.0372288 2022-03-02T06:50:04.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 134 16 57697 57681 26.8993 32.9375 0.0141321 0.0371534 2022-03-02T06:50:05.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 135 16 57844 57828 26.7681 9.1875 0.00368905 0.0370901 2022-03-02T06:50:06.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 136 16 57844 57828 26.5713 0 - 0.0370901 2022-03-02T06:50:07.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 137 16 57844 57828 26.3774 0 - 0.0370901 2022-03-02T06:50:08.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 138 16 57844 57828 26.1863 0 - 0.0370901 2022-03-02T06:50:09.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 139 16 57844 57828 25.9979 0 - 0.0370901 2022-03-02T06:50:10.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:50:10.554428+0000 min lat: 0.00139821 max lat: 11.1076 avg lat: 0.0370901 2022-03-02T06:50:10.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:50:10.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 140 16 57844 57828 25.8122 0 - 0.0370901 2022-03-02T06:50:11.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 141 16 57844 57828 25.6291 0 - 0.0370901 2022-03-02T06:50:12.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 142 16 57844 57828 25.4487 0 - 0.0370901 2022-03-02T06:50:13.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 143 16 57844 57828 25.2707 0 - 0.0370901 2022-03-02T06:50:14.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 144 16 57844 57828 25.0952 0 - 0.0370901 2022-03-02T06:50:15.505 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:15.499294 I | op-osd: OSD orchestration status for node smithi136-lv-2-data-0mp4gf is "orchestrating" 2022-03-02T06:50:15.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 145 16 57844 57828 24.9221 0 - 0.0370901 2022-03-02T06:50:15.848 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:15.842496 I | op-osd: OSD orchestration status for PVC smithi136-lv-2-data-0mp4gf is "orchestrating" 2022-03-02T06:50:16.559 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 146 16 57844 57828 24.7515 0 - 0.0370901 2022-03-02T06:50:17.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 147 16 57844 57828 24.5831 0 - 0.0370901 2022-03-02T06:50:18.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 148 16 57844 57828 24.417 0 - 0.0370901 2022-03-02T06:50:19.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 149 16 57844 57828 24.2531 0 - 0.0370901 2022-03-02T06:50:20.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 150 16 58113 58097 24.2035 1.12083 0.0193911 0.0413072 2022-03-02T06:50:21.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 151 16 58680 58664 24.2779 35.4375 0.0347136 0.0411805 2022-03-02T06:50:22.144 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:22.139177 I | op-osd: OSD orchestration status for PVC smithi136-lv-2-data-0mp4gf is "completed" 2022-03-02T06:50:22.144 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:22.139204 I | op-osd: creating OSD 0 on PVC "smithi136-lv-2-data-0mp4gf" 2022-03-02T06:50:22.145 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:22.139214 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-2-data-0mp4gf" 2022-03-02T06:50:22.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 152 16 59142 59126 24.3081 28.875 0.0268983 0.0411301 2022-03-02T06:50:23.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 153 16 59611 59595 24.3408 29.3125 0.0331391 0.0410738 2022-03-02T06:50:24.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 154 16 60072 60056 24.3698 28.8125 0.0264683 0.0410242 2022-03-02T06:50:25.207 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:25.202281 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-02T06:50:25.207 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:25.202299 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:50:25.345 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:25.338011 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:50:25.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 155 16 60616 60600 24.4319 34 0.0299315 0.0409225 2022-03-02T06:50:25.580 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:25.575678 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:50:25.658 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:25.653399 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2... 2022-03-02T06:50:26.560 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 156 16 61147 61131 24.488 33.1875 0.0186373 0.04083 2022-03-02T06:50:27.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 157 16 61719 61703 24.5597 35.75 0.0171451 0.0407093 2022-03-02T06:50:28.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 158 16 62278 62262 24.6254 34.9375 0.0510637 0.0405993 2022-03-02T06:50:29.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 159 16 62827 62811 24.6863 34.3125 0.0335422 0.040501 2022-03-02T06:50:30.562 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:50:30.556523+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.0404428 2022-03-02T06:50:30.562 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:50:30.562 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 160 16 63310 63294 24.7206 30.1875 0.0122704 0.0404428 2022-03-02T06:50:31.136 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' 3 '!=' 4 ']' 2022-03-02T06:50:31.137 INFO:teuthology.orchestra.run.smithi136.stdout:waiting for osd to come back up 2022-03-02T06:50:31.137 INFO:teuthology.orchestra.run.smithi136.stderr:+ echo 'waiting for osd to come back up' 2022-03-02T06:50:31.139 INFO:teuthology.orchestra.run.smithi136.stderr:++ toolbox ceph osd stat 2022-03-02T06:50:31.139 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-02T06:50:31.139 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f3 -d ' ' 2022-03-02T06:50:31.513 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:50:31.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 161 16 63824 63808 24.7666 32.125 0.0481274 0.0403681 2022-03-02T06:50:32.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 162 16 64257 64241 24.7808 27.0625 0.022083 0.0403251 2022-03-02T06:50:32.714 INFO:teuthology.orchestra.run.smithi136.stderr:+ num_osd=3 2022-03-02T06:50:32.715 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 30s 2022-03-02T06:50:33.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 163 16 64750 64734 24.8177 30.8125 0.0274797 0.0402836 2022-03-02T06:50:34.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 164 16 65253 65237 24.8581 31.4375 0.0676744 0.040219 2022-03-02T06:50:35.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 165 16 65823 65807 24.9233 35.625 0.026106 0.0401139 2022-03-02T06:50:36.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 166 16 66348 66332 24.9708 32.8125 0.0505477 0.040034 2022-03-02T06:50:37.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 167 16 66876 66860 25.0189 33 0.0324615 0.0399618 2022-03-02T06:50:38.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 168 16 67274 67258 25.018 24.875 0.0354786 0.0399631 2022-03-02T06:50:39.561 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 169 16 67860 67844 25.0867 36.625 0.0338706 0.0398537 2022-03-02T06:50:40.562 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 170 16 68396 68380 25.1361 33.5 0.0103708 0.0397746 2022-03-02T06:50:41.562 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 171 16 68924 68908 25.1821 33 0.0323647 0.0397023 2022-03-02T06:50:42.562 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 172 16 69420 69404 25.2158 31 0.016818 0.0396491 2022-03-02T06:50:43.562 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 173 16 69884 69868 25.2377 29 0.0368067 0.0396122 2022-03-02T06:50:44.562 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 174 16 70075 70059 25.1612 11.9375 0.00761122 0.0395571 2022-03-02T06:50:45.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 175 16 70075 70059 25.0175 0 - 0.0395571 2022-03-02T06:50:46.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 176 16 70075 70059 24.8753 0 - 0.0395571 2022-03-02T06:50:47.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 177 16 70075 70059 24.7348 0 - 0.0395571 2022-03-02T06:50:47.946 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:47.941931 I | ceph-cluster-controller: detected ceph image version: "17.0.0-10916 quincy" 2022-03-02T06:50:47.947 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:47.941955 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-02T06:50:47.947 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:47.941964 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10916 quincy", pursuing 2022-03-02T06:50:47.960 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:47.955989 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:50:48.178 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:48.174266 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:50:48.179 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:48.174507 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:50:48.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 178 16 70075 70059 24.5958 0 - 0.0395571 2022-03-02T06:50:48.638 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:48.634130 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10916 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2" 2022-03-02T06:50:49.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 179 16 70075 70059 24.4584 0 - 0.0395571 2022-03-02T06:50:49.711 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:49.690407 I | op-mon: start running mons 2022-03-02T06:50:49.916 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:49.908086 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:50:50.044 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:50.039682 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:50:50.127 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:50.119332 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:50:50.127 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:50.119496 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:50:50.514 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:50.510018 I | op-mon: targeting the mon count 1 2022-03-02T06:50:50.518 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:50.514708 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:50:50.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:50:50.559291+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.0395571 2022-03-02T06:50:50.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:50:50.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 180 16 70075 70059 24.3225 0 - 0.0395571 2022-03-02T06:50:50.971 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:50.967154 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-02T06:50:51.059 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:50.967186 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:50:51.305 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:51.301224 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-02T06:50:51.306 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:51.301244 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:50:51.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 181 16 70075 70059 24.1882 0 - 0.0395571 2022-03-02T06:50:51.772 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:51.767612 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-02T06:50:51.772 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:51.767659 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:50:52.101 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:52.097343 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-02T06:50:52.102 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:52.097370 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:50:52.448 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:52.443483 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-02T06:50:52.448 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:52.443517 I | op-config: deleting "log file" option from the mon configuration database 2022-03-02T06:50:52.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 182 16 70075 70059 24.0553 0 - 0.0395571 2022-03-02T06:50:52.804 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:52.799139 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-02T06:50:52.804 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:52.799173 I | op-mon: checking for basic quorum with existing mons 2022-03-02T06:50:53.091 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:53.086728 I | op-mon: mon "a" endpoint is [v2:10.103.147.119:3300,v1:10.103.147.119:6789] 2022-03-02T06:50:53.112 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:53.107452 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.103.147.119:6789"]}] data:a=10.103.147.119:6789 mapping:{"node":{"a":{"Name":"smithi136","Hostname":"smithi136","Address":"172.21.15.136"}}} maxMonId:0] 2022-03-02T06:50:53.118 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:53.114678 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-02T06:50:53.120 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:53.116435 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-02T06:50:53.215 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:53.210851 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-02T06:50:53.222 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:53.217480 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-02T06:50:53.223 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:53.217501 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:50:53.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 183 16 70177 70161 23.9587 0.708333 0.0195125 0.0417314 2022-03-02T06:50:53.662 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:53.657533 I | op-mon: mons running: [a] 2022-03-02T06:50:54.379 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:54.374250 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:50:54.379 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:54.374275 I | op-mon: mons created: 1 2022-03-02T06:50:54.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 184 16 70614 70598 23.9769 27.3125 0.0245215 0.0416985 2022-03-02T06:50:55.212 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:55.208191 I | op-mon: waiting for mon quorum with [a] 2022-03-02T06:50:55.297 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:55.292790 I | op-mon: mons running: [a] 2022-03-02T06:50:55.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 185 16 71035 71019 23.9895 26.3125 0.0268483 0.0416647 2022-03-02T06:50:56.345 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:56.314420 I | op-mon: Monitors in quorum: [a] 2022-03-02T06:50:56.346 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:56.314446 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-02T06:50:56.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 186 16 71443 71427 23.9976 25.5 0.0699467 0.041661 2022-03-02T06:50:56.942 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:56.938186 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-02T06:50:57.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 187 16 71927 71911 24.031 30.25 0.0218856 0.0416048 2022-03-02T06:50:58.133 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:58.126708 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-02T06:50:58.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 188 16 72440 72424 24.0737 32.0625 0.0222783 0.0415311 2022-03-02T06:50:58.934 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:50:58.928090 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-02T06:50:59.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 189 16 72893 72877 24.0961 28.3125 0.0276982 0.0414911 2022-03-02T06:51:00.443 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:00.437744 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-02T06:51:00.444 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:00.437768 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-02T06:51:00.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 190 16 73445 73429 24.1509 34.5 0.0200271 0.0413949 2022-03-02T06:51:01.170 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:01.166421 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-02T06:51:01.563 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 191 16 73958 73942 24.1923 32.0625 0.0174205 0.0413273 2022-03-02T06:51:02.063 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:02.033012 I | cephclient: successfully enabled msgr2 protocol 2022-03-02T06:51:02.071 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:02.033075 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-02T06:51:02.071 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:02.033086 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-02T06:51:02.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 192 16 74378 74362 24.203 26.25 0.0370467 0.0413068 2022-03-02T06:51:02.709 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:02.704904 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-02T06:51:02.714 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' 3 '!=' 4 ']' 2022-03-02T06:51:02.714 INFO:teuthology.orchestra.run.smithi136.stdout:waiting for osd to come back up 2022-03-02T06:51:02.715 INFO:teuthology.orchestra.run.smithi136.stderr:+ echo 'waiting for osd to come back up' 2022-03-02T06:51:02.716 INFO:teuthology.orchestra.run.smithi136.stderr:++ toolbox ceph osd stat 2022-03-02T06:51:02.716 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-02T06:51:02.716 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f3 -d ' ' 2022-03-02T06:51:03.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 193 16 74798 74782 24.2135 26.25 0.0364698 0.0412567 2022-03-02T06:51:03.607 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:51:03.607 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:03.603477 I | op-mgr: start running mgr 2022-03-02T06:51:03.614 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:03.606545 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-02T06:51:04.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 194 16 75221 75205 24.225 26.4375 0.0325314 0.0412704 2022-03-02T06:51:05.251 INFO:teuthology.orchestra.run.smithi136.stderr:+ num_osd=3 2022-03-02T06:51:05.251 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 30s 2022-03-02T06:51:05.430 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.426686 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-02T06:51:05.436 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.432887 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-02T06:51:05.441 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.436527 I | op-mgr: setting services to point to mgr "a" 2022-03-02T06:51:05.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 195 16 75760 75744 24.2735 33.6875 0.0201137 0.0411894 2022-03-02T06:51:05.700 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.696515 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-02T06:51:05.705 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.701133 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-02T06:51:05.761 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.756593 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-02T06:51:05.764 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.759784 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi136-lv-1" 2022-03-02T06:51:05.764 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.759814 I | op-osd: OSD PVC "smithi136-lv-1-data-04lnm9" already exists 2022-03-02T06:51:05.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.759826 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi136-lv-3" 2022-03-02T06:51:05.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.759835 I | op-osd: OSD PVC "smithi136-lv-3-data-0c469g" already exists 2022-03-02T06:51:05.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.759843 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi136-lv-4" 2022-03-02T06:51:05.766 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.759854 I | op-osd: OSD PVC "smithi136-lv-4-data-07m48t" already exists 2022-03-02T06:51:05.766 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.759863 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi136-lv-2" 2022-03-02T06:51:05.766 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.759874 I | op-osd: OSD PVC "smithi136-lv-2-data-0mp4gf" already exists 2022-03-02T06:51:05.791 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.776663 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-1-data-04lnm9" 2022-03-02T06:51:05.791 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.776689 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-3-data-0c469g" 2022-03-02T06:51:05.791 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.776695 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-4-data-07m48t" 2022-03-02T06:51:05.792 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.776699 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-2-data-0mp4gf" 2022-03-02T06:51:05.792 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.776703 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-02T06:51:05.792 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:05.776707 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-02T06:51:06.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 196 16 76076 76060 24.2504 19.75 0.0599923 0.0412239 2022-03-02T06:51:07.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 197 16 76531 76515 24.2717 28.4375 0.030822 0.0411919 2022-03-02T06:51:07.733 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:07.723394 I | op-mgr: successful modules: prometheus 2022-03-02T06:51:07.742 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:07.738543 I | op-mgr: successful modules: dashboard 2022-03-02T06:51:07.754 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:07.750655 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-02T06:51:08.214 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:08.209124 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-02T06:51:08.218 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:08.214962 I | op-osd: updating OSD 0 on PVC "smithi136-lv-2-data-0mp4gf" 2022-03-02T06:51:08.218 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:08.214988 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-2-data-0mp4gf" 2022-03-02T06:51:08.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 198 16 76924 76908 24.2731 24.5625 0.0508765 0.0411891 2022-03-02T06:51:08.573 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:08.568803 I | op-mgr: successful modules: balancer 2022-03-02T06:51:09.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 199 16 77336 77320 24.2805 25.75 0.0401724 0.0411729 2022-03-02T06:51:09.814 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:09.809940 I | op-mgr: successful modules: orchestrator modules 2022-03-02T06:51:10.220 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:10.216524 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-02T06:51:10.265 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:10.260734 I | op-osd: updating OSD 1 on PVC "smithi136-lv-1-data-04lnm9" 2022-03-02T06:51:10.265 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:10.260764 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-1-data-04lnm9" 2022-03-02T06:51:10.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:51:10.561362+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.0411697 2022-03-02T06:51:10.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:51:10.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 200 16 77501 77485 24.2107 10.3125 0.0507723 0.0411697 2022-03-02T06:51:11.523 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:11.519481 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-02T06:51:11.531 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:11.525269 I | op-osd: updating OSD 2 on PVC "smithi136-lv-3-data-0c469g" 2022-03-02T06:51:11.533 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:11.525346 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-3-data-0c469g" 2022-03-02T06:51:11.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 201 16 77501 77485 24.0902 0 - 0.0411697 2022-03-02T06:51:12.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 202 16 77501 77485 23.971 0 - 0.0411697 2022-03-02T06:51:12.747 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:12.743275 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-02T06:51:12.752 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:12.749257 I | op-osd: updating OSD 3 on PVC "smithi136-lv-4-data-07m48t" 2022-03-02T06:51:12.753 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:12.749279 I | op-osd: OSD will have its main bluestore block on "smithi136-lv-4-data-07m48t" 2022-03-02T06:51:13.432 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:13.427708 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-02T06:51:13.433 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:51:13.427740 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-02T06:51:13.564 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 203 16 77501 77485 23.8529 0 - 0.0411697 2022-03-02T06:51:14.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 204 16 77501 77485 23.736 0 - 0.0411697 2022-03-02T06:51:15.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 205 16 77707 77691 23.683 2.575 0.0681 0.0422148 2022-03-02T06:51:16.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 206 16 78024 78008 23.6642 19.8125 0.00990526 0.0422483 2022-03-02T06:51:17.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 207 16 78347 78331 23.6474 20.1875 0.0193975 0.0422777 2022-03-02T06:51:18.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 208 16 78665 78649 23.6292 19.875 0.0254551 0.0423094 2022-03-02T06:51:19.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 209 16 78869 78853 23.5772 12.75 0.0213941 0.0424029 2022-03-02T06:51:20.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 210 16 79183 79167 23.5583 19.625 0.0426089 0.0424358 2022-03-02T06:51:21.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 211 16 79491 79475 23.5379 19.25 0.0665598 0.0424749 2022-03-02T06:51:22.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 212 16 79680 79664 23.4826 11.8125 0.439491 0.0425209 2022-03-02T06:51:23.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 213 16 79976 79960 23.4592 18.5 0.0475733 0.042613 2022-03-02T06:51:24.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 214 16 80269 80253 23.4351 18.3125 0.0274506 0.0426618 2022-03-02T06:51:25.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 215 16 80423 80407 23.3709 9.625 0.0147474 0.0427727 2022-03-02T06:51:26.565 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 216 16 80760 80744 23.3602 21.0625 0.0513426 0.0427976 2022-03-02T06:51:27.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 217 16 81079 81063 23.3444 19.9375 0.0788422 0.0428254 2022-03-02T06:51:28.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 218 16 81392 81376 23.3271 19.5625 0.00888094 0.042855 2022-03-02T06:51:29.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 219 16 81614 81598 23.2839 13.875 0.0645696 0.0429392 2022-03-02T06:51:30.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:51:30.563463+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.0429637 2022-03-02T06:51:30.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:51:30.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 220 16 81936 81920 23.2695 20.125 0.0277872 0.0429637 2022-03-02T06:51:31.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 221 16 82146 82130 23.2236 13.125 0.00718556 0.0429831 2022-03-02T06:51:32.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 222 16 82392 82376 23.1883 15.375 0.021034 0.0431159 2022-03-02T06:51:33.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 223 16 82690 82674 23.1678 18.625 0.0400557 0.0431516 2022-03-02T06:51:34.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 224 16 82992 82976 23.1486 18.875 0.0871163 0.0431868 2022-03-02T06:51:35.252 INFO:teuthology.orchestra.run.smithi136.stdout:waiting for osd to come back up 2022-03-02T06:51:35.252 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' 3 '!=' 4 ']' 2022-03-02T06:51:35.253 INFO:teuthology.orchestra.run.smithi136.stderr:+ echo 'waiting for osd to come back up' 2022-03-02T06:51:35.253 INFO:teuthology.orchestra.run.smithi136.stderr:++ toolbox ceph osd stat 2022-03-02T06:51:35.253 INFO:teuthology.orchestra.run.smithi136.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-02T06:51:35.253 INFO:teuthology.orchestra.run.smithi136.stderr:++ cut -f3 -d ' ' 2022-03-02T06:51:35.399 INFO:teuthology.orchestra.run.smithi136.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-02T06:51:35.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 225 16 83302 83286 23.1318 19.375 0.0315101 0.0432186 2022-03-02T06:51:36.172 INFO:teuthology.orchestra.run.smithi136.stderr:+ num_osd=4 2022-03-02T06:51:36.173 INFO:teuthology.orchestra.run.smithi136.stderr:+ sleep 30s 2022-03-02T06:51:36.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 226 16 83582 83566 23.1069 17.5 0.0260587 0.0432676 2022-03-02T06:51:37.566 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 227 16 83890 83874 23.0899 19.25 0.0256569 0.0432961 2022-03-02T06:51:38.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 228 16 84183 84167 23.069 18.3125 0.0432014 0.0433381 2022-03-02T06:51:39.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 229 16 84307 84291 23.0021 7.75 0.00624882 0.043362 2022-03-02T06:51:40.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 230 16 84307 84291 22.9021 0 - 0.043362 2022-03-02T06:51:41.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 231 16 84567 84551 22.8733 8.125 0.113173 0.0437073 2022-03-02T06:51:42.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 232 16 84806 84790 22.839 14.9375 0.0383635 0.0437738 2022-03-02T06:51:43.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 233 16 85061 85045 22.8094 15.9375 0.0266894 0.0438313 2022-03-02T06:51:44.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 234 16 85140 85124 22.733 4.9375 0.0150458 0.0438393 2022-03-02T06:51:45.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 235 16 85145 85129 22.6376 0.3125 1.78488 0.0439421 2022-03-02T06:51:46.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 236 16 85207 85191 22.5581 3.875 0.00731984 0.0441683 2022-03-02T06:51:47.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 237 16 85223 85207 22.4672 1 0.0289239 0.0444148 2022-03-02T06:51:48.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 238 15 85490 85475 22.4432 16.75 0.0210466 0.0445444 2022-03-02T06:51:49.567 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 239 16 85814 85798 22.4337 20.1875 0.0272156 0.0445637 2022-03-02T06:51:50.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:51:50.565532+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.044591 2022-03-02T06:51:50.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:51:50.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 240 16 86125 86109 22.4212 19.4375 0.0201132 0.044591 2022-03-02T06:51:51.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 241 16 86417 86401 22.4039 18.25 0.0726866 0.0446137 2022-03-02T06:51:52.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 242 16 86698 86682 22.3839 17.5625 0.0281461 0.0446615 2022-03-02T06:51:53.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 243 15 87004 86989 22.3707 19.1875 0.0402397 0.0446915 2022-03-02T06:51:54.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 244 15 87163 87148 22.3198 9.9375 0.0466703 0.0447842 2022-03-02T06:51:55.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 245 16 87488 87472 22.3113 20.25 0.0755245 0.0448104 2022-03-02T06:51:56.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 246 16 87804 87788 22.3009 19.75 0.0695593 0.0448311 2022-03-02T06:51:57.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 247 16 88124 88108 22.2916 20 0.0129826 0.0448488 2022-03-02T06:51:58.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 248 16 88443 88427 22.2821 19.9375 0.0249567 0.0448687 2022-03-02T06:51:59.568 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 249 16 88756 88740 22.2711 19.5625 0.0134947 0.0448876 2022-03-02T06:52:00.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 250 16 89064 89048 22.2591 19.25 0.0387438 0.0449148 2022-03-02T06:52:01.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 251 16 89219 89203 22.209 9.6875 0.0942138 0.0450161 2022-03-02T06:52:02.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 252 16 89521 89505 22.1957 18.875 0.147833 0.0450421 2022-03-02T06:52:03.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 253 16 89810 89794 22.1794 18.0625 0.0257617 0.0450759 2022-03-02T06:52:04.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 254 16 90101 90085 22.1637 18.1875 0.0355046 0.0451092 2022-03-02T06:52:05.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 255 16 90428 90412 22.1569 20.4375 0.0365839 0.0451249 2022-03-02T06:52:06.173 INFO:teuthology.orchestra.run.smithi136.stderr:+ '[' 4 '!=' 4 ']' 2022-03-02T06:52:06.175 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-02T06:52:06.187 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch status 2022-03-02T06:52:06.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 256 16 90656 90640 22.126 14.25 0.0435908 0.0451812 2022-03-02T06:52:07.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 257 16 90960 90944 22.1138 19 0.019633 0.0452098 2022-03-02T06:52:08.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 258 16 91181 91165 22.0816 13.8125 0.0146466 0.0452313 2022-03-02T06:52:09.569 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 259 16 91304 91288 22.0261 7.6875 0.0920488 0.0453901 2022-03-02T06:52:10.570 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:52:10.567517+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.0454253 2022-03-02T06:52:10.570 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:52:10.570 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 260 16 91581 91565 22.0079 17.3125 0.0550216 0.0454253 2022-03-02T06:52:11.573 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 261 16 91728 91712 21.9588 9.1875 0.0872719 0.0455292 2022-03-02T06:52:12.571 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 262 16 91983 91967 21.9357 15.9375 0.0338706 0.0455743 2022-03-02T06:52:12.596 INFO:teuthology.orchestra.run.smithi136.stdout:Backend: rook 2022-03-02T06:52:12.596 INFO:teuthology.orchestra.run.smithi136.stdout:Available: Yes 2022-03-02T06:52:12.646 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ps 2022-03-02T06:52:13.543 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-02T06:52:13.544 INFO:teuthology.orchestra.run.smithi136.stdout:crashcollector.crash smithi136 running (7m) 0s ago 7m - - 623b85caee6c 2022-03-02T06:52:13.544 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.a smithi136 running (9m) 0s ago 9m - - 623b85caee6c 2022-03-02T06:52:13.544 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (9m) 0s ago 9m - - 623b85caee6c 2022-03-02T06:52:13.545 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (110s) 0s ago 110s - - 623b85caee6c 2022-03-02T06:52:13.545 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (7m) 0s ago 7m - - 623b85caee6c 2022-03-02T06:52:13.545 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (7m) 0s ago 7m - - 623b85caee6c 2022-03-02T06:52:13.545 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (7m) 0s ago 7m - - 623b85caee6c 2022-03-02T06:52:13.571 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 263 16 92237 92221 21.9126 15.875 0.0400415 0.0456235 2022-03-02T06:52:13.604 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls 2022-03-02T06:52:14.571 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 264 16 92482 92466 21.8876 15.3125 0.0216252 0.0456753 2022-03-02T06:52:14.631 INFO:teuthology.orchestra.run.smithi136.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-02T06:52:14.632 INFO:teuthology.orchestra.run.smithi136.stdout:crash 1/1 0s ago 7m * 2022-03-02T06:52:14.632 INFO:teuthology.orchestra.run.smithi136.stdout:mgr 1/1 0s ago 9m count:1 2022-03-02T06:52:14.632 INFO:teuthology.orchestra.run.smithi136.stdout:mon 1/1 0s ago 9m count:1 2022-03-02T06:52:14.632 INFO:teuthology.orchestra.run.smithi136.stdout:osd 8 0s ago 7m 2022-03-02T06:52:14.633 INFO:teuthology.orchestra.run.smithi136.stdout:osd.all-available-devices 0 0s ago - * 2022-03-02T06:52:14.691 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch host ls 2022-03-02T06:52:15.571 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 265 16 92744 92728 21.8668 16.375 0.0421901 0.0457227 2022-03-02T06:52:16.337 INFO:teuthology.orchestra.run.smithi136.stdout:HOST ADDR LABELS STATUS 2022-03-02T06:52:16.337 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 172.21.15.136/smithi136 2022-03-02T06:52:16.338 INFO:teuthology.orchestra.run.smithi136.stdout:1 hosts in cluster 2022-03-02T06:52:16.353 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch device ls 2022-03-02T06:52:16.571 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 266 16 92914 92898 21.8246 10.625 0.0238393 0.0457612 2022-03-02T06:52:17.572 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 267 16 92914 92898 21.7428 0 - 0.0457612 2022-03-02T06:52:17.915 INFO:teuthology.orchestra.run.smithi136.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-02T06:52:17.916 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-02T06:52:17.916 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-02T06:52:17.917 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-02T06:52:17.917 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-02T06:52:17.985 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch apply rgw foo 2022-03-02T06:52:18.572 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 268 16 93166 93150 21.7205 7.875 0.0812275 0.046028 2022-03-02T06:52:19.572 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 269 16 93418 93402 21.6983 15.75 0.0322024 0.0460748 2022-03-02T06:52:19.590 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:19.587382 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-02T06:52:19.593 INFO:teuthology.orchestra.run.smithi136.stdout:Created 2022-03-02T06:52:19.627 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch apply mds foo 2022-03-02T06:52:20.572 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 270 16 93653 93637 21.6723 14.6875 0.0335908 0.0461287 2022-03-02T06:52:20.819 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:20.813681 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:52:21.572 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 271 16 93874 93858 21.6433 13.8125 0.112382 0.046193 2022-03-02T06:52:21.620 INFO:teuthology.orchestra.run.smithi136.stdout:Created 2022-03-02T06:52:21.645 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch apply rbd-mirror 2022-03-02T06:52:21.659 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:21.620723 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-02T06:52:22.405 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:22.400036 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T06:52:22.447 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:22.444253 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:52:22.572 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 272 16 94097 94081 21.6149 13.9375 0.0850101 0.0462468 2022-03-02T06:52:23.167 INFO:teuthology.orchestra.run.smithi136.stdout:Success 2022-03-02T06:52:23.430 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:23.188875 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:52:23.432 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch apply nfs foo --port 12777 2022-03-02T06:52:23.433 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:23.412792 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:52:23.487 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:23.472474 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:52:23.487 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:23.473166 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:52:23.573 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 273 16 94290 94274 21.5799 12.0625 0.075915 0.0463239 2022-03-02T06:52:24.279 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:24.266491 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:52:24.359 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:24.356613 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-02T06:52:24.365 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:24.357199 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-02T06:52:24.366 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:24.357219 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-02T06:52:24.573 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 274 16 94459 94443 21.5397 10.5625 0.0852028 0.0464071 2022-03-02T06:52:25.573 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 275 16 94624 94608 21.4989 10.3125 0.0719157 0.0465017 2022-03-02T06:52:25.975 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:25.972885 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-02T06:52:26.313 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:26.310231 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi136": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T06:52:26.525 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:26.520251 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi136": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T06:52:26.573 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 276 16 94775 94759 21.4552 9.4375 0.125144 0.046593 2022-03-02T06:52:27.574 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 277 16 94972 94956 21.4221 12.3125 0.0888516 0.0466698 2022-03-02T06:52:28.574 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 278 16 95101 95085 21.374 8.0625 0.00714763 0.0467008 2022-03-02T06:52:28.744 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:28.742327 I | ceph-file-controller: creating filesystem "foo" 2022-03-02T06:52:29.263 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-02T06:52:29.264 INFO:teuthology.orchestra.run.smithi136.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-02T06:52:29.290 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2022-03-02T06:52:29.303 INFO:tasks.radosbench:joining radosbench (timing out after 11100s) 2022-03-02T06:52:29.303 INFO:teuthology.orchestra.run:waiting for 11100 2022-03-02T06:52:29.574 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 279 16 95101 95085 21.2974 0 - 0.0467008 2022-03-02T06:52:30.574 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:52:30.572550+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.0469975 2022-03-02T06:52:30.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:52:30.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 280 16 95321 95305 21.2705 6.875 0.0288291 0.0469975 2022-03-02T06:52:31.574 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 281 16 95539 95523 21.2432 13.625 0.0649284 0.0470539 2022-03-02T06:52:32.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 282 16 95804 95788 21.2266 16.5625 0.0791859 0.047097 2022-03-02T06:52:33.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 283 16 96026 96010 21.2007 13.875 0.0688855 0.047154 2022-03-02T06:52:34.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 284 16 96209 96193 21.1663 11.4375 0.0115742 0.0472332 2022-03-02T06:52:35.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 285 16 96449 96433 21.1446 15 0.0574166 0.0472785 2022-03-02T06:52:36.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 286 16 96681 96665 21.1214 14.5 0.0468669 0.047333 2022-03-02T06:52:37.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 287 16 96938 96922 21.1038 16.0625 0.0568603 0.0473735 2022-03-02T06:52:38.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 288 16 97202 97186 21.0878 16.5 0.0161723 0.047394 2022-03-02T06:52:39.260 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:39.257428 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.meta" 2022-03-02T06:52:39.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 289 16 97433 97417 21.0648 14.4375 0.0310773 0.0474547 2022-03-02T06:52:40.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 290 16 97702 97686 21.0501 16.8125 0.0228157 0.0474961 2022-03-02T06:52:41.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 291 16 97896 97880 21.0194 12.125 0.0489172 0.0475603 2022-03-02T06:52:42.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 292 16 98148 98132 21.0014 15.75 0.0814794 0.0476069 2022-03-02T06:52:43.575 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 293 16 98429 98413 20.9896 17.5625 0.0416429 0.0476331 2022-03-02T06:52:44.083 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:44.073851 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.non-ec" 2022-03-02T06:52:44.096 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:44.083912 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-metadata" 2022-03-02T06:52:44.096 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:44.090709 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.control" 2022-03-02T06:52:44.104 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:44.098386 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.index" 2022-03-02T06:52:44.105 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:44.100478 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.log" 2022-03-02T06:52:44.576 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 294 16 98616 98600 20.958 11.6875 0.0641465 0.0477027 2022-03-02T06:52:45.578 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 295 16 98828 98812 20.9317 13.25 0.053554 0.047763 2022-03-02T06:52:46.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 296 16 99108 99092 20.9201 17.5 0.0393564 0.0477918 2022-03-02T06:52:47.019 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:47.017197 I | cephclient: creating replicated pool foo.rgw.meta succeeded 2022-03-02T06:52:47.026 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:47.023361 I | cephclient: creating replicated pool foo-metadata succeeded 2022-03-02T06:52:47.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 297 16 99299 99283 20.8898 11.9375 0.0551813 0.0478575 2022-03-02T06:52:48.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 298 16 99563 99547 20.8751 16.5 0.076868 0.047892 2022-03-02T06:52:49.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 299 16 99819 99803 20.8588 16 0.0224679 0.0479291 2022-03-02T06:52:50.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:52:50.577343+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.047988 2022-03-02T06:52:50.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:52:50.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 300 16 100032 100016 20.8336 13.3125 0.0218776 0.047988 2022-03-02T06:52:51.370 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:51.365171 I | cephclient: creating replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-02T06:52:51.391 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:51.384887 I | cephclient: creating replicated pool foo.rgw.log succeeded 2022-03-02T06:52:51.413 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:51.405397 I | cephclient: creating replicated pool foo.rgw.control succeeded 2022-03-02T06:52:51.420 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:51.418142 I | cephclient: creating replicated pool foo.rgw.buckets.index succeeded 2022-03-02T06:52:51.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:51.488149 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:52:51.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:51.488180 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:52:51.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 301 16 100289 100273 20.8178 16.0625 0.0703328 0.0480236 2022-03-02T06:52:52.471 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:52.465671 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:52:52.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 302 16 100536 100520 20.7999 15.4375 0.0539812 0.0480656 2022-03-02T06:52:53.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 303 16 100768 100752 20.7791 14.5 0.02131 0.0481146 2022-03-02T06:52:54.130 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:54.127367 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-data0" 2022-03-02T06:52:54.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 304 16 100983 100967 20.755 13.4375 0.0345298 0.0481674 2022-03-02T06:52:54.991 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:54.988113 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:52:55.579 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 305 16 101241 101225 20.7398 16.125 0.0264405 0.0482071 2022-03-02T06:52:56.488 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:56.481954 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:52:56.489 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:56.481979 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:52:56.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 306 16 101501 101485 20.7251 16.25 0.0530628 0.0482396 2022-03-02T06:52:56.605 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:56.603008 I | cephclient: creating replicated pool foo-data0 succeeded 2022-03-02T06:52:56.610 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:56.603043 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-02T06:52:57.202 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:57.199786 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:52:57.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 307 15 101675 101660 20.6932 10.9375 0.0686084 0.0483128 2022-03-02T06:52:58.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 308 16 101854 101838 20.6622 11.125 0.0512579 0.0483876 2022-03-02T06:52:58.651 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:58.648791 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi136": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T06:52:59.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 309 16 102095 102079 20.6441 15.0625 0.112119 0.04843 2022-03-02T06:52:59.718 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:52:59.711905 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-02T06:53:00.503 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:00.501318 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-02T06:53:00.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 310 16 102324 102308 20.6236 14.3125 0.0523756 0.0484752 2022-03-02T06:53:01.178 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:01.174808 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:53:01.268 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:01.261504 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:53:01.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 311 16 102543 102527 20.6013 13.6875 0.0631043 0.0485302 2022-03-02T06:53:02.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 312 16 102788 102772 20.5844 15.3125 0.0875993 0.0485694 2022-03-02T06:53:03.243 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:03.235626 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-02T06:53:03.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 313 16 103028 103012 20.5665 15 0.036382 0.0486119 2022-03-02T06:53:03.880 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:03.878183 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:53:04.075 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:04.069590 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:53:04.076 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:04.069617 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:53:04.096 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:04.093953 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-02T06:53:04.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 314 16 103225 103209 20.5402 12.3125 0.0355068 0.0486718 2022-03-02T06:53:04.785 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:04.782732 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:53:05.371 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:05.362362 I | op-mds: setting mds config flags 2022-03-02T06:53:05.371 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:05.362392 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-02T06:53:05.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 315 16 103393 103377 20.5084 10.5 0.079647 0.0487462 2022-03-02T06:53:06.449 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:06.446048 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-02T06:53:06.458 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:06.453570 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-02T06:53:06.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 316 16 103592 103576 20.4828 12.4375 0.109025 0.0488046 2022-03-02T06:53:07.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 317 16 103803 103787 20.4598 13.1875 0.0760043 0.0488636 2022-03-02T06:53:08.256 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:08.253602 I | op-mds: setting mds config flags 2022-03-02T06:53:08.256 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:08.253666 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-02T06:53:08.567 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:08.562891 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:53:08.580 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 318 16 103993 103977 20.4328 11.875 0.0311833 0.0489305 2022-03-02T06:53:08.954 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:08.951782 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:53:09.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 319 16 104228 104212 20.4148 14.6875 0.0904055 0.0489716 2022-03-02T06:53:10.033 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:10.030697 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-02T06:53:10.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:53:10.579342+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.0490288 2022-03-02T06:53:10.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:53:10.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 320 16 104436 104420 20.3916 13 0.142767 0.0490288 2022-03-02T06:53:11.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 321 16 104655 104639 20.3707 13.6875 0.0216164 0.0490751 2022-03-02T06:53:11.731 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:11.728420 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:53:12.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 322 16 104912 104896 20.3573 16.0625 0.0201851 0.0491092 2022-03-02T06:53:12.855 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:12.847635 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:53:12.855 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:12.847662 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:53:13.506 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:13.503224 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:53:13.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 323 16 105105 105089 20.3316 12.0625 0.0728328 0.049169 2022-03-02T06:53:14.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 324 16 105268 105252 20.3003 10.1875 0.145064 0.0492446 2022-03-02T06:53:15.581 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 325 16 105460 105444 20.2748 12 0.0393473 0.0493108 2022-03-02T06:53:16.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 326 16 105720 105704 20.2624 16.25 0.00680624 0.0493295 2022-03-02T06:53:17.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 327 16 105963 105947 20.2469 15.1875 0.0555859 0.0493775 2022-03-02T06:53:18.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 328 16 106192 106176 20.2288 14.3125 0.0277889 0.0494165 2022-03-02T06:53:19.314 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:19.304167 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:53:19.356 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:19.353552 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:53:19.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 329 16 106400 106384 20.2068 13 0.01562 0.0494721 2022-03-02T06:53:20.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 330 16 106659 106643 20.1947 16.1875 0.0852301 0.0495043 2022-03-02T06:53:21.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 331 16 106859 106843 20.1714 12.5 0.128801 0.0495604 2022-03-02T06:53:22.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 332 16 107030 107014 20.1428 10.6875 0.0225869 0.049632 2022-03-02T06:53:23.226 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:23.223864 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:53:23.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 333 16 107273 107257 20.128 15.1875 0.0245686 0.0496701 2022-03-02T06:53:24.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 334 16 107529 107513 20.1156 16 0.0794043 0.0496995 2022-03-02T06:53:24.661 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:24.659401 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:53:24.662 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:24.659422 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:53:25.355 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:25.353137 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:53:25.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 335 16 107740 107724 20.0949 13.1875 0.216611 0.049748 2022-03-02T06:53:26.582 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 336 16 107808 107792 20.0478 4.25 0.0253119 0.0498394 2022-03-02T06:53:27.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 337 16 108036 108020 20.0306 14.25 0.0514095 0.0499111 2022-03-02T06:53:28.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 338 16 108268 108252 20.0142 14.5 0.0669384 0.0499554 2022-03-02T06:53:29.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 339 16 108554 108538 20.0079 17.875 0.0476248 0.0499656 2022-03-02T06:53:30.182 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:30.180057 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:53:30.391 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:30.384142 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:53:30.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:2022-03-02T06:53:30.581336+0000 min lat: 0.00139821 max lat: 15.7233 avg lat: 0.0500217 2022-03-02T06:53:30.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-02T06:53:30.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 340 16 108758 108742 19.9865 12.75 0.0520546 0.0500217 2022-03-02T06:53:31.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 341 16 109052 109036 19.9818 18.375 0.0725594 0.0500236 2022-03-02T06:53:32.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 342 16 109311 109295 19.9707 16.1875 0.288941 0.0500494 2022-03-02T06:53:33.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 343 16 109553 109537 19.9566 15.125 0.0212809 0.0500898 2022-03-02T06:53:33.809 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:33.804067 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:53:34.331 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:34.326538 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:53:34.331 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:34.326564 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:53:34.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 344 16 109804 109788 19.9441 15.6875 0.295026 0.0501136 2022-03-02T06:53:34.920 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:34.917689 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:53:35.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 345 16 109934 109918 19.9099 8.125 0.056067 0.0501934 2022-03-02T06:53:36.583 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 346 16 110214 110198 19.9029 17.5 0.0108293 0.0502327 2022-03-02T06:53:37.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 347 16 110551 110535 19.9063 21.0625 0.020751 0.0502257 2022-03-02T06:53:38.029 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:38.024370 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:53:38.195 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:38.192915 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:53:38.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 348 16 110893 110877 19.9105 21.375 0.0426315 0.050216 2022-03-02T06:53:39.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 349 16 111211 111195 19.9104 19.875 0.0308604 0.0502139 2022-03-02T06:53:40.337 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:40.332480 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:53:40.486 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:40.483688 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:53:40.487 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:40.483719 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:53:40.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 350 16 111576 111560 19.9186 22.8125 0.0164133 0.0501949 2022-03-02T06:53:41.151 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:41.147472 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:53:41.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 351 16 111864 111848 19.9132 18 0.0098816 0.0501973 2022-03-02T06:53:42.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 352 16 112057 112041 19.8909 12.0625 0.0738518 0.050265 2022-03-02T06:53:43.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 353 16 112350 112334 19.8864 18.3125 0.015261 0.050277 2022-03-02T06:53:44.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 354 16 112677 112661 19.8879 20.4375 0.0378575 0.0502702 2022-03-02T06:53:45.110 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:45.107765 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:53:45.439 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:45.436292 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:53:45.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 355 16 113051 113035 19.8978 23.375 0.0831471 0.0502457 2022-03-02T06:53:46.584 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 356 16 113443 113427 19.9107 24.5 0.120521 0.0502154 2022-03-02T06:53:47.411 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:47.403047 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:53:47.543 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:47.540470 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:53:47.543 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:47.540487 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:53:47.585 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 357 16 113611 113595 19.8843 10.5 0.00859802 0.050281 2022-03-02T06:53:48.116 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:48.114379 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:53:48.585 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 358 16 113907 113891 19.8804 18.5 0.0775225 0.0502866 2022-03-02T06:53:49.585 INFO:tasks.radosbench.radosbench.a.smithi136.stdout: 359 16 114020 114004 19.8447 7.0625 0.0117209 0.0503096 2022-03-02T06:53:50.585 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Total time run: 360.031 2022-03-02T06:53:50.585 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Total writes made: 114327 2022-03-02T06:53:50.586 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Write size: 65536 2022-03-02T06:53:50.586 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Object size: 65536 2022-03-02T06:53:50.586 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Bandwidth (MB/sec): 19.8468 2022-03-02T06:53:50.586 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Stddev Bandwidth: 16.1861 2022-03-02T06:53:50.587 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Max bandwidth (MB/sec): 86.1875 2022-03-02T06:53:50.587 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Min bandwidth (MB/sec): 0 2022-03-02T06:53:50.587 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Average IOPS: 317 2022-03-02T06:53:50.587 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Stddev IOPS: 258.989 2022-03-02T06:53:50.588 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Max IOPS: 1379 2022-03-02T06:53:50.588 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Min IOPS: 0 2022-03-02T06:53:50.588 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Average Latency(s): 0.0503786 2022-03-02T06:53:50.588 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Stddev Latency(s): 0.333163 2022-03-02T06:53:50.589 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Max latency(s): 15.7233 2022-03-02T06:53:50.589 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Min latency(s): 0.00139821 2022-03-02T06:53:50.673 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Cleaning up (deleting benchmark objects) 2022-03-02T06:53:51.111 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:51.109067 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:53:51.762 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:51.760284 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:53:53.811 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:53.809014 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:53:53.928 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:53.919354 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:53:53.928 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:53.919378 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:53:54.506 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:54.494401 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:53:57.404 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:57.401397 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:53:58.695 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:53:58.691262 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:54:00.878 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:00.875614 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:54:01.009 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:01.005106 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:54:01.012 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:01.005133 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:54:01.608 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:01.596631 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:54:04.245 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:04.241347 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:54:06.816 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:06.813542 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:54:08.740 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:08.738262 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:54:10.643 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:10.638365 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:54:10.644 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:10.638405 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:54:11.177 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:11.172804 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:54:15.875 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:15.873138 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:54:21.635 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:21.627596 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:54:23.866 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:23.861363 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:54:24.083 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:24.078266 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:54:24.086 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:24.078293 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:54:24.789 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:24.787481 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:54:27.519 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:27.516025 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:54:37.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:37.762060 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:54:39.721 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:39.716823 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:54:39.882 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:39.877374 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:54:39.882 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:39.877399 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:54:40.481 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:40.473246 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:54:45.389 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:54:45.387456 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:55:05.876 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:05.873849 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:55:08.291 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:08.289262 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:55:08.944 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:08.941966 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:55:08.945 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:08.942060 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:55:09.647 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:09.644381 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:55:12.700 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:12.696337 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:55:53.768 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:53.766139 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:55:56.335 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:56.330204 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:55:58.070 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:58.068539 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:55:58.071 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:58.068568 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:55:58.988 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:55:58.985198 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:56:02.499 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:56:02.495345 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T06:57:24.724 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:57:24.718457 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T06:57:26.460 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:57:26.455500 I | ceph-object-controller: reconciling object store deployments 2022-03-02T06:57:26.604 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:57:26.601649 I | ceph-object-controller: ceph object store gateway service running at 10.106.111.25 2022-03-02T06:57:26.615 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:57:26.601674 I | ceph-object-controller: reconciling object store pools 2022-03-02T06:57:27.235 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:57:27.227949 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-02T06:57:32.980 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 06:57:32.978023 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-02T07:00:03.040 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Removed 114327 objects 2022-03-02T07:00:03.041 INFO:tasks.radosbench.radosbench.a.smithi136.stdout:Clean up completed and total clean up time :372.366 2022-03-02T07:00:06.905 INFO:tasks.rook.ceph_manager.ceph:removing pool_name unique_pool_0 2022-03-02T07:00:06.906 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2022-03-02T07:00:08.349 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_0' removed 2022-03-02T07:00:08.372 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-03-02T07:00:08.393 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-03-02T07:00:08.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-03-02T07:00:08.482 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-02T07:00:08.496 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:00:09.321 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:00:09.321 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:52:26.000000Z", "last_refresh": "2022-03-02T07:00:09.210911Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:00:09.210911Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:00:09.210911Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:00:09.210911Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:00:09.210911Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:00:09.210911Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-02T07:00:09.210911Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:00:09.210911Z", "running": 0, "size": 1}}] 2022-03-02T07:00:09.341 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch rm mds.foo 2022-03-02T07:00:09.847 INFO:teuthology.orchestra.run.smithi136.stdout:Removed cephfilesystems/foo 2022-03-02T07:00:09.870 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch rm rbd-mirror 2022-03-02T07:00:10.310 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:10.306884 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T07:00:10.800 INFO:teuthology.orchestra.run.smithi136.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-02T07:00:10.824 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch rm rgw.foo 2022-03-02T07:00:11.520 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:11.516186 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-02T07:00:12.986 INFO:teuthology.orchestra.run.smithi136.stdout:Removed cephobjectstores/foo 2022-03-02T07:00:13.007 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:00:13.241 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:13.215996 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-02T07:00:13.988 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:13.934239 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T07:00:15.375 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:15.371828 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-02T07:00:15.376 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:15.371848 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-02T07:00:15.376 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:15.371854 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-02T07:00:15.675 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:00:15.676 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:52:26.000000Z", "last_refresh": "2022-03-02T07:00:15.049101Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:00:15.049101Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:00:15.049101Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:00:15.049101Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:00:15.049101Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:00:15.049101Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:00:15.049101Z", "running": 0, "size": 1}}] 2022-03-02T07:00:18.650 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:18.648065 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-02T07:00:19.301 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:19.298182 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-02T07:00:20.139 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:20.136529 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-02T07:00:20.139 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:20.136565 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-02T07:00:20.140 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:20.136591 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-02T07:00:21.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:21.190969 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:00:21.182+0000 7f9fc5e3e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:00:21.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:00:21.182+0000 7f9fc5e3e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:00:21.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:00:21.182+0000 7f9fc5e3e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:00:21.195 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:00:21.195 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:21.190994 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:00:21.182+0000 7f9fc5e3e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:00:21.195 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:00:21.182+0000 7f9fc5e3e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:00:21.195 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:00:21.182+0000 7f9fc5e3e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:00:21.196 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:00:21.351 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:21.348422 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-02T07:00:21.351 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:21.348453 I | ceph-file-controller: downing filesystem "foo" 2022-03-02T07:00:21.656 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:21.653764 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T07:00:22.646 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:22.643559 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:00:22.630+0000 7f01a9aca340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:00:22.647 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:00:22.630+0000 7f01a9aca340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:00:22.647 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:00:22.630+0000 7f01a9aca340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:00:22.647 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:00:22.647 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:22.643587 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:00:22.630+0000 7f01a9aca340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:00:22.648 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:00:22.630+0000 7f01a9aca340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:00:22.648 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:00:22.630+0000 7f01a9aca340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:00:22.648 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:00:24.354 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:24.352006 I | ceph-file-controller: downed filesystem "foo" 2022-03-02T07:00:25.697 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:00:26.837 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:26.835346 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-02T07:00:26.838 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:26.835366 I | cephclient: purging pool "foo-metadata" (id=5) 2022-03-02T07:00:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:00:26.990 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:52:26.000000Z", "last_refresh": "2022-03-02T07:00:26.409282Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:00:26.409282Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:00:26.409282Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:00:26.409282Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:00:26.409282Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:00:26.409282Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:00:26.409282Z", "running": 0, "size": 1}}] 2022-03-02T07:00:30.987 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:30.984467 I | cephclient: purge completed for pool "foo-metadata" 2022-03-02T07:00:31.419 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:31.412750 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-02T07:00:31.419 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:31.412773 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-02T07:00:33.453 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:33.450591 I | cephclient: purge completed for pool "foo-data0" 2022-03-02T07:00:33.453 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:33.450621 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-02T07:00:33.549 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:33.546863 E | ceph-file-controller: failed to reconcile failed to remove finalizer: failed to remove finalizer "cephfilesystem.ceph.rook.io" on "foo": Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T07:00:33.559 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:33.556647 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T07:00:34.406 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:34.403483 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-02T07:00:34.449 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:34.444866 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-02T07:00:34.455 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:34.453207 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-02T07:00:34.458 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:34.453942 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-02T07:00:34.458 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:34.454030 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-02T07:00:37.009 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:00:37.945 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:37.941528 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi136": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T07:00:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:00:37.972 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:52:26.000000Z", "last_refresh": "2022-03-02T07:00:37.614129Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:00:37.614129Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:00:37.614129Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:00:37.614129Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:00:37.614129Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:00:37.614129Z", "running": 0, "size": 1}}] 2022-03-02T07:00:38.039 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:00:38.036245 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi136": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T07:00:47.993 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:00:48.859 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:00:48.860 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:52:26.000000Z", "last_refresh": "2022-03-02T07:00:48.502263Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:00:48.502263Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:00:48.502263Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:00:48.502263Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:00:48.502263Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:00:48.502263Z", "running": 0, "size": 1}}] 2022-03-02T07:00:58.880 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:00:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:00:59.495 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:00:59.398671Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:00:59.398671Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:00:59.398671Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:00:59.398671Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:00:59.398671Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:00:59.398671Z", "running": 0, "size": 1}}] 2022-03-02T07:01:09.521 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:01:10.153 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:01:10.154 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:01:10.051761Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:01:10.051761Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:01:10.051761Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:01:10.051761Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:01:10.051761Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:01:10.051761Z", "running": 0, "size": 1}}] 2022-03-02T07:01:20.176 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:01:20.804 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:01:20.804 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:01:20.701627Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:01:20.701627Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:01:20.701627Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:01:20.701627Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:01:20.701627Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:01:20.701627Z", "running": 0, "size": 1}}] 2022-03-02T07:01:30.827 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:01:31.483 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:01:31.484 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:01:31.347160Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:01:31.347160Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:01:31.347160Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:01:31.347160Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:01:31.347160Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:01:31.347160Z", "running": 0, "size": 1}}] 2022-03-02T07:01:41.505 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:01:42.180 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:01:42.180 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:01:42.012882Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:01:42.012882Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:01:42.012882Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:01:42.012882Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:01:42.012882Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:01:42.012882Z", "running": 0, "size": 1}}] 2022-03-02T07:01:52.202 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:01:52.838 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:01:52.838 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:01:52.743451Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:01:52.743451Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:01:52.743451Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:01:52.743451Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:01:52.743451Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:01:52.743451Z", "running": 0, "size": 1}}] 2022-03-02T07:02:02.863 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:02:03.455 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:02:03.455 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:02:03.358840Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:02:03.358840Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:02:03.358840Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:02:03.358840Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:02:03.358840Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:02:03.358840Z", "running": 0, "size": 1}}] 2022-03-02T07:02:13.478 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:02:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:02:17.040 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:02:16.942145Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:02:16.942145Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:02:16.942145Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:02:16.942145Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:02:16.942145Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:02:16.942145Z", "running": 0, "size": 1}}] 2022-03-02T07:02:27.065 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:02:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:02:27.686 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:02:27.590898Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:02:27.590898Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:02:27.590898Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:02:27.590898Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:02:27.590898Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:02:27.590898Z", "running": 0, "size": 1}}] 2022-03-02T07:02:37.711 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:02:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:02:38.330 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:02:38.235894Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:02:38.235894Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:02:38.235894Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:02:38.235894Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:02:38.235894Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:02:38.235894Z", "running": 0, "size": 1}}] 2022-03-02T07:02:48.353 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:02:48.947 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:02:48.947 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:02:48.850890Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:02:48.850890Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:02:48.850890Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:02:48.850890Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:02:48.850890Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:02:48.850890Z", "running": 0, "size": 1}}] 2022-03-02T07:02:58.970 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:02:59.597 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:02:59.597 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:02:59.498635Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:02:59.498635Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:02:59.498635Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:02:59.498635Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:02:59.498635Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:02:59.498635Z", "running": 0, "size": 1}}] 2022-03-02T07:03:09.622 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:03:10.253 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:03:10.253 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:03:10.164066Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:03:10.164066Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:03:10.164066Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:03:10.164066Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:03:10.164066Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:03:10.164066Z", "running": 0, "size": 1}}] 2022-03-02T07:03:20.280 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:03:20.927 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:03:20.927 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:03:20.788974Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:03:20.788974Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:03:20.788974Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:03:20.788974Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:03:20.788974Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:03:20.788974Z", "running": 0, "size": 1}}] 2022-03-02T07:03:30.951 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:03:31.950 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:03:31.951 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:03:31.508949Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:03:31.508949Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:03:31.508949Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:03:31.508949Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:03:31.508949Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:03:31.508949Z", "running": 0, "size": 1}}] 2022-03-02T07:03:41.977 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:03:42.604 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:03:42.605 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:03:42.507839Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:03:42.507839Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:03:42.507839Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:03:42.507839Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:03:42.507839Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:03:42.507839Z", "running": 0, "size": 1}}] 2022-03-02T07:03:52.630 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:03:53.223 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:03:53.223 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:03:53.129620Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:03:53.129620Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:03:53.129620Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:03:53.129620Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:03:53.129620Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:03:53.129620Z", "running": 0, "size": 1}}] 2022-03-02T07:04:03.247 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:04:03.865 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:04:03.865 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:04:03.742748Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:04:03.742748Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:04:03.742748Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:04:03.742748Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:04:03.742748Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:04:03.742748Z", "running": 0, "size": 1}}] 2022-03-02T07:04:13.888 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:04:14.841 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:04:14.841 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:04:14.466820Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:04:14.466820Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:04:14.466820Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:04:14.466820Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:04:14.466820Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:04:14.466820Z", "running": 0, "size": 1}}] 2022-03-02T07:04:24.864 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:04:25.463 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:04:25.463 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:04:25.369923Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:04:25.369923Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:04:25.369923Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:04:25.369923Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:04:25.369923Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:04:25.369923Z", "running": 0, "size": 1}}] 2022-03-02T07:04:35.486 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:04:36.203 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:04:36.203 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:04:36.103650Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:04:36.103650Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:04:36.103650Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:04:36.103650Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:04:36.103650Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:04:36.103650Z", "running": 0, "size": 1}}] 2022-03-02T07:04:46.228 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:04:46.854 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:04:46.854 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:04:46.758458Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:04:46.758458Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:04:46.758458Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:04:46.758458Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:04:46.758458Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:04:46.758458Z", "running": 0, "size": 1}}] 2022-03-02T07:04:56.877 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:04:57.478 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:04:57.479 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:04:57.384928Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:04:57.384928Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:04:57.384928Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:04:57.384928Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:04:57.384928Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:04:57.384928Z", "running": 0, "size": 1}}] 2022-03-02T07:05:07.503 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:05:08.136 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:05:08.136 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:05:08.047654Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:05:08.047654Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:05:08.047654Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:05:08.047654Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:05:08.047654Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:05:08.047654Z", "running": 0, "size": 1}}] 2022-03-02T07:05:18.157 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:05:18.799 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:05:18.799 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:05:18.702176Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:05:18.702176Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:05:18.702176Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:05:18.702176Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:05:18.702176Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:05:18.702176Z", "running": 0, "size": 1}}] 2022-03-02T07:05:28.820 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:05:29.431 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:05:29.432 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:05:29.326001Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:05:29.326001Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:05:29.326001Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:05:29.326001Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:05:29.326001Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:05:29.326001Z", "running": 0, "size": 1}}] 2022-03-02T07:05:39.451 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:05:40.147 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:05:40.147 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:05:40.010833Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:05:40.010833Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:05:40.010833Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:05:40.010833Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:05:40.010833Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:05:40.010833Z", "running": 0, "size": 1}}] 2022-03-02T07:05:48.885 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:05:48.881463 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T07:05:49.492 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:05:49.488511 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:05:49.478+0000 7f799f2eb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:05:49.492 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:05:49.478+0000 7f799f2eb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:05:49.493 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:05:49.478+0000 7f799f2eb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:05:49.493 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:05:49.493 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:05:49.488536 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:05:49.478+0000 7f799f2eb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:05:49.493 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:05:49.478+0000 7f799f2eb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:05:49.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:05:49.478+0000 7f799f2eb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:05:49.494 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:05:50.172 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:05:50.797 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:05:50.798 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:05:50.703140Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:05:50.703140Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:05:50.703140Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:05:50.703140Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:05:50.703140Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:05:50.703140Z", "running": 0, "size": 1}}] 2022-03-02T07:06:00.820 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:06:01.437 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:06:01.437 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:06:01.338428Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:06:01.338428Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:06:01.338428Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:06:01.338428Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:06:01.338428Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:06:01.338428Z", "running": 0, "size": 1}}] 2022-03-02T07:06:11.491 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:06:12.168 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:06:12.169 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:06:12.004182Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:06:12.004182Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:06:12.004182Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:06:12.004182Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:06:12.004182Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:06:12.004182Z", "running": 0, "size": 1}}] 2022-03-02T07:06:22.192 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:06:22.846 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:06:22.846 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:06:22.748061Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:06:22.748061Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:06:22.748061Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:06:22.748061Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:06:22.748061Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:06:22.748061Z", "running": 0, "size": 1}}] 2022-03-02T07:06:32.871 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:06:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:06:33.439 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:06:33.344331Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:06:33.344331Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:06:33.344331Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:06:33.344331Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:06:33.344331Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:06:33.344331Z", "running": 0, "size": 1}}] 2022-03-02T07:06:43.464 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:06:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:06:44.132 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:06:43.950559Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:06:43.950559Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:06:43.950559Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:06:43.950559Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:06:43.950559Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:06:43.950559Z", "running": 0, "size": 1}}] 2022-03-02T07:06:54.156 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:06:54.838 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:06:54.839 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:06:54.735834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:06:54.735834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:06:54.735834Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:06:54.735834Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:06:54.735834Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:06:54.735834Z", "running": 0, "size": 1}}] 2022-03-02T07:07:04.863 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:07:05.504 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:07:05.505 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:07:05.402965Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:07:05.402965Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:07:05.402965Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:07:05.402965Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:07:05.402965Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:07:05.402965Z", "running": 0, "size": 1}}] 2022-03-02T07:07:15.527 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:07:16.180 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:07:16.181 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:07:16.083105Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:07:16.083105Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:07:16.083105Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:07:16.083105Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:07:16.083105Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:07:16.083105Z", "running": 0, "size": 1}}] 2022-03-02T07:07:26.202 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:07:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:07:26.867 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:07:26.758681Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:07:26.758681Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:07:26.758681Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:07:26.758681Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:07:26.758681Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:07:26.758681Z", "running": 0, "size": 1}}] 2022-03-02T07:07:36.891 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:07:37.499 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:07:37.500 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:07:37.404631Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:07:37.404631Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:07:37.404631Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:07:37.404631Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:07:37.404631Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:07:37.404631Z", "running": 0, "size": 1}}] 2022-03-02T07:07:47.518 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:07:48.217 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:07:48.218 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:07:48.110845Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:07:48.110845Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:07:48.110845Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:07:48.110845Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:07:48.110845Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:07:48.110845Z", "running": 0, "size": 1}}] 2022-03-02T07:07:58.241 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:07:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:07:58.853 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:07:58.758754Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:07:58.758754Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:07:58.758754Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:07:58.758754Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:07:58.758754Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:07:58.758754Z", "running": 0, "size": 1}}] 2022-03-02T07:08:08.876 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:08:09.554 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:08:09.554 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:08:09.435663Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:08:09.435663Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:08:09.435663Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:08:09.435663Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:08:09.435663Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:08:09.435663Z", "running": 0, "size": 1}}] 2022-03-02T07:08:19.579 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:08:20.208 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:08:20.209 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:08:20.120914Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:08:20.120914Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:08:20.120914Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:08:20.120914Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:08:20.120914Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:08:20.120914Z", "running": 0, "size": 1}}] 2022-03-02T07:08:30.232 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:08:30.840 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:08:30.841 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:08:30.756517Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:08:30.756517Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:08:30.756517Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:08:30.756517Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:08:30.756517Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:08:30.756517Z", "running": 0, "size": 1}}] 2022-03-02T07:08:40.865 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:08:41.516 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:08:41.516 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:08:41.401150Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:08:41.401150Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:08:41.401150Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:08:41.401150Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:08:41.401150Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:08:41.401150Z", "running": 0, "size": 1}}] 2022-03-02T07:08:51.539 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:08:52.148 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:08:52.148 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:08:52.041630Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:08:52.041630Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:08:52.041630Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:08:52.041630Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:08:52.041630Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:08:52.041630Z", "running": 0, "size": 1}}] 2022-03-02T07:09:02.175 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:09:02.841 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:09:02.841 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:09:02.727594Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:09:02.727594Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:09:02.727594Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:09:02.727594Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:09:02.727594Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:09:02.727594Z", "running": 0, "size": 1}}] 2022-03-02T07:09:12.862 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:09:13.444 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:09:13.445 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:09:13.348240Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:09:13.348240Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:09:13.348240Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:09:13.348240Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:09:13.348240Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:09:13.348240Z", "running": 0, "size": 1}}] 2022-03-02T07:09:23.471 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:09:24.133 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:09:24.133 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:09:24.038381Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:09:24.038381Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:09:24.038381Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:09:24.038381Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:09:24.038381Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:09:24.038381Z", "running": 0, "size": 1}}] 2022-03-02T07:09:34.156 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:09:34.787 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:09:34.787 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:09:34.690196Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:09:34.690196Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:09:34.690196Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:09:34.690196Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:09:34.690196Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:09:34.690196Z", "running": 0, "size": 1}}] 2022-03-02T07:09:44.808 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:09:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:09:45.492 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:09:45.352813Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:09:45.352813Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:09:45.352813Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:09:45.352813Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:09:45.352813Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:09:45.352813Z", "running": 0, "size": 1}}] 2022-03-02T07:09:55.517 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:09:56.163 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:09:56.163 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:09:56.058955Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:09:56.058955Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:09:56.058955Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:09:56.058955Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:09:56.058955Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:09:56.058955Z", "running": 0, "size": 1}}] 2022-03-02T07:10:06.181 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:10:06.834 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:10:06.834 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:10:06.732539Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:10:06.732539Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:10:06.732539Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:10:06.732539Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:10:06.732539Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:10:06.732539Z", "running": 0, "size": 1}}] 2022-03-02T07:10:16.855 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:10:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:10:17.913 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:10:17.373822Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:10:17.373822Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:10:17.373822Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:10:17.373822Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:10:17.373822Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:10:17.373822Z", "running": 0, "size": 1}}] 2022-03-02T07:10:27.939 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:10:28.648 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:10:28.648 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:10:28.547106Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:10:28.547106Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:10:28.547106Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:10:28.547106Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:10:28.547106Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:10:28.547106Z", "running": 0, "size": 1}}] 2022-03-02T07:10:38.673 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:10:39.303 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:10:39.303 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:10:39.200152Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:10:39.200152Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:10:39.200152Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:10:39.200152Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:10:39.200152Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:10:39.200152Z", "running": 0, "size": 1}}] 2022-03-02T07:10:49.327 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:10:50.011 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:10:50.012 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:10:49.843117Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:10:49.843117Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:10:49.843117Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:10:49.843117Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:10:49.843117Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:10:49.843117Z", "running": 0, "size": 1}}] 2022-03-02T07:11:00.034 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:11:00.659 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:11:00.660 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:11:00.522855Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:11:00.522855Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:11:00.522855Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:11:00.522855Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:11:00.522855Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:11:00.522855Z", "running": 0, "size": 1}}] 2022-03-02T07:11:10.684 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:11:11.323 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:11:11.481 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:11:11.234958Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:11:11.234958Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:11:11.234958Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:11:11.234958Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:11:11.234958Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:11:11.234958Z", "running": 0, "size": 1}}] 2022-03-02T07:11:21.484 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:11:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:11:22.079 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:11:21.989371Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:11:21.989371Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:11:21.989371Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:11:21.989371Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:11:21.989371Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:11:21.989371Z", "running": 0, "size": 1}}] 2022-03-02T07:11:32.102 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:11:32.779 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:11:32.779 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:11:32.681002Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:11:32.681002Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:11:32.681002Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:11:32.681002Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:11:32.681002Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:11:32.681002Z", "running": 0, "size": 1}}] 2022-03-02T07:11:42.805 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:11:43.465 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:11:43.465 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:11:43.303940Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:11:43.303940Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:11:43.303940Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:11:43.303940Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:11:43.303940Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:11:43.303940Z", "running": 0, "size": 1}}] 2022-03-02T07:11:53.491 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:11:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:11:54.151 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:11:54.015709Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:11:54.015709Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:11:54.015709Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:11:54.015709Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:11:54.015709Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:11:54.015709Z", "running": 0, "size": 1}}] 2022-03-02T07:12:04.172 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:12:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:12:04.785 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:12:04.657649Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:12:04.657649Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:12:04.657649Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:12:04.657649Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:12:04.657649Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:12:04.657649Z", "running": 0, "size": 1}}] 2022-03-02T07:12:14.808 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:12:15.859 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:12:15.859 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:12:15.758982Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:12:15.758982Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:12:15.758982Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:12:15.758982Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:12:15.758982Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:12:15.758982Z", "running": 0, "size": 1}}] 2022-03-02T07:12:25.881 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:12:29.019 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:12:29.019 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:12:28.918438Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:12:28.918438Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:12:28.918438Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:12:28.918438Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:12:28.918438Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:12:28.918438Z", "running": 0, "size": 1}}] 2022-03-02T07:12:39.040 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:12:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:12:39.650 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:12:39.554905Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:12:39.554905Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:12:39.554905Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:12:39.554905Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:12:39.554905Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:12:39.554905Z", "running": 0, "size": 1}}] 2022-03-02T07:12:49.674 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:12:50.285 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:12:50.286 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:12:50.174874Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:12:50.174874Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:12:50.174874Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:12:50.174874Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:12:50.174874Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:12:50.174874Z", "running": 0, "size": 1}}] 2022-03-02T07:13:00.308 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:13:01.112 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:13:01.113 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:13:00.917693Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:13:00.917693Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:13:00.917693Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:13:00.917693Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:13:00.917693Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:13:00.917693Z", "running": 0, "size": 1}}] 2022-03-02T07:13:11.137 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:13:11.784 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:13:11.785 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:13:11.689623Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:13:11.689623Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:13:11.689623Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:13:11.689623Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:13:11.689623Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:13:11.689623Z", "running": 0, "size": 1}}] 2022-03-02T07:13:21.809 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:13:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:13:22.541 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:13:22.286371Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:13:22.286371Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:13:22.286371Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:13:22.286371Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:13:22.286371Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:13:22.286371Z", "running": 0, "size": 1}}] 2022-03-02T07:13:32.571 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:13:33.210 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:13:33.210 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:13:33.114937Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:13:33.114937Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:13:33.114937Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:13:33.114937Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:13:33.114937Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:13:33.114937Z", "running": 0, "size": 1}}] 2022-03-02T07:13:43.234 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:13:43.952 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:13:43.953 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:13:43.749920Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:13:43.749920Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:13:43.749920Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:13:43.749920Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:13:43.749920Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:13:43.749920Z", "running": 0, "size": 1}}] 2022-03-02T07:13:54.003 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:13:54.709 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:13:54.710 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:13:54.611467Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:13:54.611467Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:13:54.611467Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:13:54.611467Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:13:54.611467Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:13:54.611467Z", "running": 0, "size": 1}}] 2022-03-02T07:14:04.736 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:14:05.331 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:14:05.332 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:14:05.236114Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:14:05.236114Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:14:05.236114Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:14:05.236114Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:14:05.236114Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:14:05.236114Z", "running": 0, "size": 1}}] 2022-03-02T07:14:15.353 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:14:15.969 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:14:15.969 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:14:15.858640Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:14:15.858640Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:14:15.858640Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:14:15.858640Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:14:15.858640Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:14:15.858640Z", "running": 0, "size": 1}}] 2022-03-02T07:14:25.991 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:14:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:14:26.614 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:14:26.507759Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:14:26.507759Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:14:26.507759Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:14:26.507759Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:14:26.507759Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:14:26.507759Z", "running": 0, "size": 1}}] 2022-03-02T07:14:36.637 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:14:37.260 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:14:37.260 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:14:37.140878Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:14:37.140878Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:14:37.140878Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:14:37.140878Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:14:37.140878Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:14:37.140878Z", "running": 0, "size": 1}}] 2022-03-02T07:14:47.286 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:14:47.914 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:14:47.915 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:14:47.754471Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:14:47.754471Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:14:47.754471Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:14:47.754471Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:14:47.754471Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:14:47.754471Z", "running": 0, "size": 1}}] 2022-03-02T07:14:57.939 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:14:58.549 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:14:58.550 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:14:58.457367Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:14:58.457367Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:14:58.457367Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:14:58.457367Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:14:58.457367Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:14:58.457367Z", "running": 0, "size": 1}}] 2022-03-02T07:15:08.573 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:15:09.193 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:15:09.194 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:15:09.101404Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:15:09.101404Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:15:09.101404Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:15:09.101404Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:15:09.101404Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:15:09.101404Z", "running": 0, "size": 1}}] 2022-03-02T07:15:19.217 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:15:19.897 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:15:19.898 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:15:19.748215Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:15:19.748215Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:15:19.748215Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:15:19.748215Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:15:19.748215Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:15:19.748215Z", "running": 0, "size": 1}}] 2022-03-02T07:15:29.922 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:15:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:15:30.524 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:15:30.424132Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:15:30.424132Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:15:30.424132Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:15:30.424132Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:15:30.424132Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:15:30.424132Z", "running": 0, "size": 1}}] 2022-03-02T07:15:40.545 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:15:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:15:41.212 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:15:41.035685Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:15:41.035685Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:15:41.035685Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:15:41.035685Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:15:41.035685Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:15:41.035685Z", "running": 0, "size": 1}}] 2022-03-02T07:15:51.233 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:15:51.987 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:15:51.988 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:15:51.891885Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:15:51.891885Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:15:51.891885Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:15:51.891885Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:15:51.891885Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:15:51.891885Z", "running": 0, "size": 1}}] 2022-03-02T07:16:02.010 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:16:02.631 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:16:02.632 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:16:02.540868Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:16:02.540868Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:16:02.540868Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:16:02.540868Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:16:02.540868Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:16:02.540868Z", "running": 0, "size": 1}}] 2022-03-02T07:16:12.654 DEBUG:teuthology.orchestra.run.smithi136:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-hdpwp -- ceph orch ls -f json 2022-03-02T07:16:13.308 INFO:teuthology.orchestra.run.smithi136.stdout: 2022-03-02T07:16:13.309 INFO:teuthology.orchestra.run.smithi136.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T07:00:37.000000Z", "last_refresh": "2022-03-02T07:16:13.223998Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:48.000000Z", "last_refresh": "2022-03-02T07:16:13.223998Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:42:22.000000Z", "last_refresh": "2022-03-02T07:16:13.223998Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "623b85caee6c61dd40928e0fca3f577e3b7785e69f1e3eb9bc50a78987d4ee02", "container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "created": "2022-03-02T06:44:41.000000Z", "last_refresh": "2022-03-02T07:16:13.223998Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-02T07:16:13.223998Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2", "last_refresh": "2022-03-02T07:16:13.223998Z", "running": 0, "size": 1}}] 2022-03-02T07:16:13.326 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-02T07:16:13.327 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-02T07:16:13.328 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-02T07:16:13.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-02T07:16:13.588 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-02T07:16:13.589 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-02T07:16:13.627 DEBUG:teuthology.orchestra.remote:smithi136:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-02T07:16:13.640 DEBUG:teuthology.orchestra.run.smithi136:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-02T07:16:13.702 INFO:teuthology.orchestra.run.smithi136.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-02T07:16:13.718 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8b0a50f4a1537dd23aa80e7b80c2449d30bc80d2/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_cb5b1a75970b7aa81700a0ab081943b3146d40d0/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-02T07:16:13.719 DEBUG:teuthology.orchestra.run.smithi136:> kubectl delete -f cluster.yaml 2022-03-02T07:16:13.897 INFO:teuthology.orchestra.run.smithi136.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-02T07:16:13.899 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:13.896522 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-02T07:16:14.420 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:14.417665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:14.657 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:14.648466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:14.658 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:14.648491 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:24.869 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:24.866780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:24.885 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:24.882470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:35.103 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:35.100947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:35.116 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:35.113606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:45.338 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:45.336086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:45.352 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:45.350284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:55.589 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:55.586562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:16:55.598 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:16:55.596338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:05.818 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:05.816156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:05.835 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:05.831927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:16.050 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:16.048000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:16.062 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:16.059849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:26.278 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:26.275619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:26.288 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:26.286264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:36.516 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:36.513476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:36.529 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:36.526993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:46.751 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:46.748415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:46.760 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:46.758315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:56.975 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:56.973304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:17:56.988 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:17:56.986733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:07.208 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:07.205949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:07.218 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:07.217170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:17.443 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:17.441160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:17.453 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:17.452018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:27.674 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:27.672150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:27.687 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:27.685747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:37.914 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:37.912474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:37.925 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:37.923809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:48.147 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:48.145161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:48.162 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:48.160551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:58.384 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:58.381894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:18:58.461 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:18:58.459454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:08.678 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:08.676312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:08.691 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:08.689181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:18.912 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:18.910795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:18.956 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:18.954507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:29.179 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:29.177306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:29.191 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:29.189403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:39.448 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:39.446655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:39.461 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:39.459416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:49.704 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:49.702439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:49.717 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:49.715468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:59.936 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:59.934118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:19:59.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:19:59.946840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:10.171 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:10.169513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:10.182 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:10.180986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:20.422 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:20.420839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:20.434 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:20.432515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:30.651 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:30.649777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:30.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:30.665657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:40.895 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:40.893267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:40.928 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:40.926495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:51.152 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:51.150535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:20:51.163 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:20:51.162358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:01.388 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:01.386572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:01.401 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:01.399640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:11.617 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:11.615896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:11.626 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:11.625233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:21.858 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:21.856803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:21.870 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:21.868677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:32.098 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:32.096200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:32.121 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:32.119945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:42.376 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:42.374446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:42.393 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:42.391824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:52.635 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:52.634231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:21:52.647 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:21:52.646110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:02.875 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:02.873552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:02.888 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:02.886832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:13.104 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:13.103034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:13.113 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:13.112612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:23.334 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:23.333365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:23.348 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:23.345214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:29.499 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:29.497479 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T07:22:30.089 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:30.086648 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:22:30.080+0000 7f8789cc1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:22:30.089 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:22:30.080+0000 7f8789cc1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:22:30.090 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:22:30.080+0000 7f8789cc1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:22:30.090 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:22:30.090 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:30.086673 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:22:30.080+0000 7f8789cc1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:22:30.090 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:22:30.080+0000 7f8789cc1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:22:30.091 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:22:30.080+0000 7f8789cc1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:22:30.091 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:22:33.562 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:33.561279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:33.574 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:33.573319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:43.796 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:43.794513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:43.807 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:43.806724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:54.031 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:54.029891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:22:54.044 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:22:54.043113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:04.267 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:04.265857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:04.278 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:04.277105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:14.523 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:14.522215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:14.537 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:14.536503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:24.755 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:24.754620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:24.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:24.763966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:34.985 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:34.984501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:35.000 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:34.998224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:45.221 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:45.220223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:45.234 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:45.232207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:55.466 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:55.464647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:55.481 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:55.480481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:23:55.651 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:23:55.650681 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 07:23:55.640689652 +0000 UTC m=+2576.950490298 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T07:24:05.702 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:05.701071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:05.713 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:05.712834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:15.953 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:15.951943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:15.964 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:15.963173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:26.223 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:26.221555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:26.233 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:26.232931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:36.471 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:36.470222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:36.484 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:36.483766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:46.700 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:46.699112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:46.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:46.728179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:56.946 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:56.945114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:24:56.961 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:24:56.960136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:07.181 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:07.180209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:07.193 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:07.192141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:17.415 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:17.413999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:17.425 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:17.424386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:27.644 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:27.643677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:27.659 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:27.658474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:37.880 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:37.879535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:37.893 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:37.892767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:48.115 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:48.113703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:48.126 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:48.125945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:58.369 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:58.368559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:25:58.379 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:25:58.378838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:08.604 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:08.602823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:08.615 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:08.614312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:18.836 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:18.835615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:18.849 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:18.849211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:29.071 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:29.070516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:29.086 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:29.085961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:39.311 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:39.310134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:39.323 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:39.322287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:49.547 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:49.546138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:49.559 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:49.559190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:59.784 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:59.783099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:26:59.797 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:26:59.794642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:10.018 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:10.017163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:10.030 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:10.029387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:20.255 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:20.254399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:20.269 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:20.268890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:30.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:30.493662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:30.507 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:30.505991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:40.725 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:40.723871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:40.736 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:40.735286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:50.957 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:50.955861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:27:50.968 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:27:50.967484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:01.186 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:01.185755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:01.197 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:01.197027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:11.415 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:11.414442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:11.424 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:11.423837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:21.672 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:21.672205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:21.683 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:21.683346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:31.905 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:31.905024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:31.915 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:31.915112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:42.142 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:42.141732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:42.152 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:42.151175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:52.383 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:52.382513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:28:52.395 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:28:52.394432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:02.610 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:02.609165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:02.619 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:02.618699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:12.836 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:12.835847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:12.847 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:12.846862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:23.067 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:23.066744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:23.078 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:23.077690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:33.333 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:33.333097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:33.353 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:33.352549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:43.570 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:43.569827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:43.627 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:43.627032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:53.844 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:53.843772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:29:53.856 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:29:53.856173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:04.075 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:04.074352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:04.087 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:04.086264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:14.309 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:14.308382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:14.318 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:14.317854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:24.543 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:24.541760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:24.552 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:24.551269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:34.776 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:34.775890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:34.788 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:34.788152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:45.010 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:45.009555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:45.021 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:45.021148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:55.247 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:55.246223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:30:55.259 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:30:55.259043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:05.484 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:05.482901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:05.495 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:05.495036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:15.717 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:15.716262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:15.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:15.729271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:25.952 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:25.951519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:25.964 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:25.963436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:36.556 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:36.554903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:36.570 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:36.569698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:46.796 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:46.795481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:46.807 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:46.807122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:57.067 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:57.066506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:31:57.079 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:31:57.079537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:07.499 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:07.499227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:07.511 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:07.511453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:17.733 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:17.732490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:17.783 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:17.782984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:28.005 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:28.004942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:28.018 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:28.018575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:38.606 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:38.605294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:38.630 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:38.629912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:48.846 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:48.845530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:48.858 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:48.858011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:59.086 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:59.086143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:32:59.098 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:32:59.097994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:09.551 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:09.550497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:09.563 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:09.562854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:19.779 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:19.778527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:19.794 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:19.790851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:30.010 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:30.009565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:30.024 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:30.023408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:40.239 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:40.238784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:40.252 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:40.252406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:50.474 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:50.473407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:33:50.485 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:33:50.484865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:00.706 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:00.705445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:00.719 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:00.718804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:10.977 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:10.976322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:10.991 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:10.990854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:21.252 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:21.252402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:21.263 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:21.263412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:31.485 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:31.484345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:31.497 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:31.496696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:41.739 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:41.738333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:41.746 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:41.746223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:51.985 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:51.984885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:34:51.999 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:34:51.999061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:02.224 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:02.223523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:02.236 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:02.235896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:12.456 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:12.456218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:12.505 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:12.504589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:22.723 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:22.722797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:22.734 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:22.734375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:32.957 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:32.956614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:32.969 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:32.968407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:43.233 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:43.232454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:43.243 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:43.242837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:53.465 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:53.464289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:35:53.476 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:35:53.476030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:03.699 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:03.698552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:03.709 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:03.709706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:14.085 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:14.085360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:14.098 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:14.096878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:24.309 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:24.309087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:24.319 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:24.319142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:34.601 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:34.600491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:34.614 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:34.614113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:44.857 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:44.856500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:44.869 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:44.868524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:55.088 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:55.087623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:36:55.100 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:36:55.099788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:05.324 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:05.323781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:05.335 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:05.334757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:15.559 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:15.558219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:15.570 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:15.569743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:25.792 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:25.792037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:25.803 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:25.802946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:36.028 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:36.027554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:36.039 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:36.038881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:46.347 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:46.346552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:46.358 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:46.357599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:56.583 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:56.583006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:37:56.595 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:37:56.595034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:06.812 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:06.811677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:06.825 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:06.825502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:17.311 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:17.311079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:17.322 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:17.322395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:27.580 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:27.580276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:27.592 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:27.592401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:37.822 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:37.821683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:37.834 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:37.833727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:48.276 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:48.275932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:48.290 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:48.289744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:58.511 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:58.510704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:38:58.526 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:38:58.525751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:08.656 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:08.655734 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 07:39:08.646760171 +0000 UTC m=+3489.956560819 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T07:39:08.754 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:08.754298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:08.768 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:08.766582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:10.092 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:10.092349 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T07:39:10.732 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:10.731156 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:39:10.722+0000 7f8540379340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:39:10.732 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:39:10.722+0000 7f8540379340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:39:10.732 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:39:10.722+0000 7f8540379340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:39:10.733 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:39:10.733 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:10.731178 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:39:10.722+0000 7f8540379340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:39:10.733 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:39:10.722+0000 7f8540379340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:39:10.733 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:39:10.722+0000 7f8540379340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:39:10.733 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:39:19.044 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:19.044136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:19.298 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:19.297487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:29.517 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:29.516692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:29.529 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:29.529333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:39.767 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:39.766542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:39.780 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:39.779941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:50.189 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:50.188541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:39:50.261 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:39:50.260836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:00.482 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:00.481922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:00.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:00.494561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:10.718 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:10.717986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:10.731 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:10.730855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:20.952 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:20.952101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:20.966 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:20.965557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:31.189 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:31.189164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:31.201 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:31.200948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:41.429 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:41.428914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:41.441 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:41.441302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:51.875 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:51.874262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:40:51.883 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:40:51.883222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:02.109 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:02.108321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:02.120 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:02.119939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:12.352 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:12.352165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:12.364 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:12.363469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:22.837 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:22.836299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:22.903 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:22.848305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:33.070 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:33.069798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:33.126 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:33.125932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:43.344 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:43.343283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:43.352 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:43.351745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:53.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:53.728656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:41:53.742 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:41:53.741673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:04.053 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:04.052912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:04.061 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:04.060822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:14.277 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:14.276951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:14.292 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:14.292225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:24.747 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:24.746899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:24.763 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:24.760686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:34.982 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:34.981450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:34.997 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:34.996299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:45.228 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:45.227669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:45.240 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:45.237385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:55.548 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:55.547912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:42:55.565 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:42:55.563902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:05.795 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:05.794289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:05.806 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:05.805742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:16.027 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:16.026407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:16.039 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:16.038486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:26.258 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:26.258086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:26.270 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:26.269971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:36.488 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:36.487568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:36.502 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:36.501114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:46.726 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:46.725543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:46.737 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:46.736854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:56.965 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:56.964300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:43:56.973 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:43:56.972808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:07.197 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:07.196286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:07.210 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:07.209130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:17.437 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:17.437010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:17.478 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:17.477658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:27.707 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:27.706215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:27.719 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:27.719363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:37.944 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:37.944133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:37.956 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:37.955700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:48.174 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:48.173380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:48.187 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:48.186956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:58.410 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:58.409170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:44:58.422 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:44:58.421855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:08.643 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:08.642211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:08.655 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:08.654511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:18.877 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:18.876660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:18.885 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:18.884688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:29.112 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:29.111985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:29.124 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:29.124690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:39.351 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:39.350789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:39.361 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:39.361486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:49.582 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:49.581919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:49.592 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:49.591552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:59.855 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:59.855192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:45:59.868 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:45:59.867887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:10.088 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:10.088426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:10.098 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:10.098117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:20.319 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:20.318807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:20.331 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:20.330702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:30.547 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:30.547188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:30.562 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:30.561591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:40.783 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:40.783265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:40.794 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:40.794742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:51.036 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:51.035566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:46:51.049 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:46:51.048583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:01.383 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:01.382346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:01.609 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:01.609738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:11.829 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:11.828451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:11.839 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:11.839639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:22.063 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:22.062710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:22.075 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:22.075580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:32.527 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:32.526554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:32.539 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:32.539631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:42.758 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:42.758000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:42.768 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:42.768386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:52.991 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:52.990728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:47:53.003 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:47:53.002826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:03.224 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:03.223667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:03.235 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:03.235706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:13.463 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:13.462557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:13.474 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:13.474246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:23.692 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:23.691454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:23.706 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:23.706360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:34.015 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:34.014942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:34.024 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:34.024149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:44.260 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:44.259824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:44.273 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:44.273122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:54.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:54.494273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:48:54.515 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:48:54.515785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:04.954 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:04.954277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:04.969 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:04.968557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:15.189 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:15.188360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:15.202 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:15.202370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:25.424 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:25.423921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:25.436 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:25.436232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:35.969 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:35.967478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:35.979 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:35.979775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:46.203 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:46.202619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:46.216 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:46.216091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:56.435 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:56.434517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:49:56.455 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:49:56.455365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:06.876 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:06.875834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:06.887 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:06.887920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:17.114 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:17.114407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:17.123 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:17.123736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:27.368 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:27.368121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:27.379 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:27.379076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:37.791 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:37.790940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:37.836 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:37.835276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:48.062 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:48.061801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:48.071 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:48.071286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:58.292 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:58.291947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:50:58.303 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:50:58.303783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:08.527 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:08.527519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:08.541 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:08.540862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:18.767 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:18.758065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:18.769 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:18.769268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:18.984 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:18.984501 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 07:51:18.973261597 +0000 UTC m=+4220.283062268 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T07:51:28.995 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:28.995100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:29.008 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:29.008146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:39.231 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:39.230743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:39.244 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:39.244598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:49.468 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:49.467514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:49.482 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:49.482169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:59.700 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:59.700429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:51:59.712 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:51:59.712138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:09.938 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:09.937610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:09.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:09.948763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:20.204 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:20.203855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:20.218 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:20.218039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:30.450 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:30.449197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:30.474 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:30.473718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:40.714 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:40.714312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:40.727 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:40.726899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:50.955 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:50.954881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:52:50.966 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:52:50.966435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:01.188 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:01.188033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:01.205 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:01.201829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:11.511 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:11.510927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:11.520 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:11.520680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:21.744 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:21.743882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:21.813 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:21.811032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:32.035 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:32.033304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:32.048 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:32.046174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:42.454 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:42.453968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:42.486 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:42.486023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:52.756 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:52.755619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:53:52.767 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:53:52.766990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:03.018 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:03.017669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:03.031 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:03.029968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:13.354 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:13.353919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:13.366 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:13.366722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:23.591 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:23.590483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:23.601 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:23.600992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:33.824 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:33.823550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:33.836 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:33.836478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:44.186 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:44.186468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:44.196 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:44.196543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:54.421 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:54.420854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:54:54.433 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:54:54.433640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:04.651 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:04.651333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:04.665 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:04.665300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:14.890 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:14.888967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:14.902 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:14.902615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:25.126 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:25.125420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:25.144 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:25.144206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:35.370 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:35.369602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:35.383 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:35.382978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:45.608 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:45.606996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:45.620 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:45.620171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:50.744 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:50.744719 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T07:55:51.333 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:51.333345 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:55:51.323+0000 7fce079b2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:55:51.334 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:55:51.323+0000 7fce079b2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:55:51.334 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:55:51.323+0000 7fce079b2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:55:51.334 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:55:51.335 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:51.333371 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T07:55:51.323+0000 7fce079b2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T07:55:51.335 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:55:51.323+0000 7fce079b2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T07:55:51.335 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T07:55:51.323+0000 7fce079b2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T07:55:51.335 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T07:55:55.940 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:55.939383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:55:55.956 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:55:55.956761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:06.180 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:06.179674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:06.189 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:06.189407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:16.571 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:16.571069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:16.582 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:16.582147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:26.872 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:26.871751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:26.884 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:26.884848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:37.107 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:37.106856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:37.119 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:37.119719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:47.344 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:47.344135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:47.357 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:47.356957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:57.591 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:57.591270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:56:57.604 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:56:57.604685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:07.830 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:07.830185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:07.843 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:07.843037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:18.096 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:18.095802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:18.108 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:18.107704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:28.343 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:28.343361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:28.357 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:28.357651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:38.580 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:38.579368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:38.594 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:38.594265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:48.846 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:48.846199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:48.859 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:48.859189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:59.121 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:59.120637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:57:59.136 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:57:59.136349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:09.356 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:09.355897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:09.368 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:09.368668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:19.583 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:19.583000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:19.601 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:19.596898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:29.826 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:29.826205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:29.835 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:29.835870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:40.057 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:40.056968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:40.071 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:40.071228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:50.424 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:50.424120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:58:50.434 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:58:50.434160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:00.655 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:00.654827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:00.668 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:00.668550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:10.889 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:10.888851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:10.902 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:10.901755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:21.119 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:21.119368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:21.132 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:21.132254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:31.355 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:31.355322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:31.367 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:31.367655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:41.591 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:41.590883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:41.607 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:41.607395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:52.105 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:52.104310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T07:59:52.135 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 07:59:52.135591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:02.385 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:02.385341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:02.425 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:02.425277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:12.665 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:12.665321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:12.678 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:12.677978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:22.904 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:22.904446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:22.917 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:22.917539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:33.139 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:33.138722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:33.152 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:33.151716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:43.375 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:43.374548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:43.418 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:43.417674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:53.733 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:53.731270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:00:53.746 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:00:53.746283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:03.969 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:03.968679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:03.979 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:03.978462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:14.242 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:14.242019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:14.252 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:14.251780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:24.542 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:24.541820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:24.553 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:24.553188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:34.773 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:34.772246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:34.787 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:34.787528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:45.008 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:45.007624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:45.021 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:45.020957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:55.460 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:55.459427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:01:55.474 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:01:55.474297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:05.701 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:05.700518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:05.713 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:05.713123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:15.936 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:15.935945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:15.951 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:15.950739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:26.225 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:26.225146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:26.236 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:26.236547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:36.464 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:36.464503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:36.478 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:36.478510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:46.704 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:46.704123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:46.713 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:46.713515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:56.977 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:56.977624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:02:56.989 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:02:56.988504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:07.207 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:07.207048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:07.217 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:07.217420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:17.439 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:17.438569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:17.450 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:17.450402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:27.902 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:27.900376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:27.938 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:27.937907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:38.220 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:38.219769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:38.235 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:38.234601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:48.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:48.493571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:48.506 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:48.506677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:58.831 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:58.830895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:03:58.847 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:03:58.844985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:09.084 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:09.084448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:09.094 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:09.094063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:19.313 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:19.313316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:19.325 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:19.324857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:29.545 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:29.545194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:29.554 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:29.554797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:39.773 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:39.773488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:39.788 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:39.787636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:50.008 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:50.008642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:04:50.020 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:04:50.019876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:00.237 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:00.237293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:00.246 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:00.246323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:10.470 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:10.469647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:10.483 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:10.482789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:20.713 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:20.712576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:20.724 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:20.724118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:30.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:30.947603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:30.958 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:30.958717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:31.207 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:31.207747 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 08:05:31.197290178 +0000 UTC m=+5072.507090807 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T08:05:41.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:41.193619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:41.203 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:41.202256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:51.426 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:51.425981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:05:51.468 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:05:51.468122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:01.695 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:01.694752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:01.707 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:01.706762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:11.929 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:11.929072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:11.939 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:11.939130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:22.163 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:22.163619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:22.176 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:22.176358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:32.395 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:32.395538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:32.412 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:32.412665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:42.632 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:42.632402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:42.643 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:42.643738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:52.867 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:52.866589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:06:52.879 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:06:52.878868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:03.099 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:03.099501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:03.112 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:03.111690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:13.329 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:13.328864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:13.341 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:13.341785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:23.569 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:23.568555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:23.579 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:23.579902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:33.807 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:33.802293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:33.814 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:33.814613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:44.037 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:44.036533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:44.046 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:44.046394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:54.296 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:54.295742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:07:54.320 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:07:54.320020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:04.534 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:04.533796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:04.548 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:04.545473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:14.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:14.764784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:14.778 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:14.777506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:24.998 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:24.997819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:25.012 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:25.012065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:35.263 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:35.262391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:35.276 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:35.276762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:45.495 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:45.494782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:45.507 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:45.507663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:55.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:55.728060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:08:55.742 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:08:55.742300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:05.965 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:05.964639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:05.976 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:05.976255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:16.200 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:16.199619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:16.214 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:16.212701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:26.439 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:26.439172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:26.459 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:26.458061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:36.684 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:36.683566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:36.695 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:36.695110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:46.921 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:46.920955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:46.933 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:46.932463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:57.155 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:57.154309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:09:57.187 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:09:57.187493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:07.425 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:07.424820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:07.442 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:07.442178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:17.672 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:17.671441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:17.684 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:17.684558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:27.907 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:27.907062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:27.920 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:27.920343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:38.145 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:38.143891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:38.156 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:38.156486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:48.385 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:48.384361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:48.398 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:48.397618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:58.626 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:58.625340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:10:58.638 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:10:58.636471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:08.857 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:08.856459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:08.869 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:08.868661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:19.089 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:19.088394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:19.100 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:19.100051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:29.321 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:29.320867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:29.332 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:29.332023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:39.549 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:39.548372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:39.560 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:39.559790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:49.777 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:49.776623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:11:49.789 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:11:49.788792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:00.026 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:00.025336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:00.040 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:00.039873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:10.255 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:10.254832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:10.266 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:10.265613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:20.490 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:20.489531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:20.502 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:20.501714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:30.737 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:30.736336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:30.747 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:30.746819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:31.343 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:31.342825 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T08:12:32.073 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:32.072598 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T08:12:32.064+0000 7fcc63f3e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T08:12:32.073 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:12:32.064+0000 7fcc63f3e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T08:12:32.074 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:12:32.064+0000 7fcc63f3e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T08:12:32.074 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T08:12:32.074 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:32.072622 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T08:12:32.064+0000 7fcc63f3e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T08:12:32.075 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:12:32.064+0000 7fcc63f3e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T08:12:32.075 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:12:32.064+0000 7fcc63f3e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T08:12:32.075 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T08:12:40.962 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:40.961288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:40.977 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:40.976609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:51.196 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:51.195195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:12:51.240 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:12:51.239303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:01.462 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:01.461574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:01.475 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:01.474236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:11.692 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:11.692256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:11.705 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:11.704642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:21.933 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:21.928641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:22.073 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:21.940548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:32.163 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:32.162451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:32.174 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:32.173585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:42.439 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:42.438953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:42.447 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:42.447372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:52.666 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:52.665766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:13:52.677 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:13:52.677285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:02.903 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:02.902858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:02.917 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:02.916699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:13.137 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:13.136943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:13.148 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:13.147909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:23.377 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:23.372920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:23.387 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:23.385467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:33.613 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:33.612357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:33.627 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:33.627202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:43.857 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:43.855897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:43.870 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:43.869343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:54.091 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:54.090404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:14:54.101 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:14:54.100682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:04.323 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:04.322837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:04.337 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:04.336470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:14.563 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:14.563164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:14.575 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:14.575125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:24.800 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:24.799810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:24.813 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:24.812560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:35.039 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:35.038800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:35.060 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:35.059807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:45.297 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:45.296487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:45.308 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:45.308164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:55.540 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:55.539338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:15:55.556 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:15:55.555556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:05.776 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:05.775412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:05.788 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:05.787642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:16.006 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:16.005673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:16.051 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:16.049867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:16.051 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:16.049896 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:26.271 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:26.270688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:26.287 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:26.285980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:36.506 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:36.505911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:36.518 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:36.517924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:46.740 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:46.739610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:46.751 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:46.750707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:56.972 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:56.971201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:16:56.986 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:16:56.985816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:07.208 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:07.207317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:07.221 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:07.220064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:17.497 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:17.496234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:17.510 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:17.509658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:27.731 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:27.730103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:27.742 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:27.741489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:37.967 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:37.966425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:37.979 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:37.978812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:48.200 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:48.199606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:48.213 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:48.212299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:58.439 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:58.437987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:17:58.446 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:17:58.446469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:08.673 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:08.672626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:08.686 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:08.685856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:18.911 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:18.910273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:18.923 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:18.922530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:29.159 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:29.158294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:29.171 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:29.171475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:39.395 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:39.394754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:39.407 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:39.404943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:49.636 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:49.635132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:49.653 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:49.653112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:59.875 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:59.873954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:18:59.888 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:18:59.887322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:10.106 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:10.105595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:10.116 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:10.115591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:20.361 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:20.360399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:20.372 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:20.371787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:30.633 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:30.633188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:30.646 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:30.646474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:40.901 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:40.901763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:40.912 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:40.912822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:51.127 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:51.128093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:19:51.188 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:19:51.188507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:01.406 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:01.406855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:01.419 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:01.419961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:11.639 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:11.640546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:11.646 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:11.648067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:21.863 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:21.864679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:21.909 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:21.910333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:32.162 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:32.163525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:32.172 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:32.174359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:42.424 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:42.425986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:42.437 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:42.439451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:52.678 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:52.679845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:20:52.688 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:20:52.690616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:02.909 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:02.911168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:02.951 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:02.922912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:13.138 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:13.140987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:13.148 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:13.150109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:23.369 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:23.371506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:23.380 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:23.382712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:33.628 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:33.630808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:33.635 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:33.638571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:43.861 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:43.863704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:43.873 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:43.876907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:54.091 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:54.095016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:21:54.104 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:21:54.107806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:04.328 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:04.331726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:04.340 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:04.344720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:14.568 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:14.571949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:14.582 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:14.586534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:24.803 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:24.806622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:24.815 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:24.819605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:35.043 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:35.046504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:35.055 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:35.059584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:45.281 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:45.284644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:45.293 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:45.298245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:55.517 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:55.521356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:22:55.531 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:22:55.535444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:05.753 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:05.757547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:05.764 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:05.769511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:15.984 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:15.988723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:15.997 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:16.002750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:26.226 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:26.231262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:26.239 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:26.244603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:36.469 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:36.474259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:36.480 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:36.486043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:46.719 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:46.724468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:46.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:46.734873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:46.769 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:46.774667 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 08:23:46.759555982 +0000 UTC m=+6168.069356661 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T08:23:56.949 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:56.955101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:23:56.963 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:23:56.969084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:07.205 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:07.210745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:07.217 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:07.223123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:17.436 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:17.442247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:17.449 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:17.456017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:27.668 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:27.673952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:27.679 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:27.685424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:37.919 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:37.925634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:37.930 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:37.936758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:48.151 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:48.156896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:48.162 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:48.168983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:58.382 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:58.388487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:24:58.392 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:24:58.399505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:08.613 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:08.619519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:08.626 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:08.632686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:18.846 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:18.853390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:18.863 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:18.870204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:29.084 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:29.091450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:29.097 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:29.104160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:39.320 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:39.327064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:39.334 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:39.340882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:49.554 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:49.561556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:49.567 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:49.574817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:59.788 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:59.795375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:25:59.811 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:25:59.817907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:10.031 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:10.037141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:10.041 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:10.049090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:20.263 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:20.270289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:20.275 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:20.283363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:30.499 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:30.507015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:30.507 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:30.515446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:40.726 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:40.733989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:40.738 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:40.746443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:50.968 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:50.975916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:26:50.981 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:26:50.989319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:01.201 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:01.209014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:01.212 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:01.220741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:11.431 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:11.439559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:11.446 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:11.454997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:21.663 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:21.671393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:21.675 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:21.684305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:31.903 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:31.911323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:31.917 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:31.924225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:42.143 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:42.151923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:42.203 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:42.211155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:52.425 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:52.433139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:27:52.437 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:27:52.445845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:02.654 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:02.662807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:02.665 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:02.674079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:12.893 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:12.902088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:12.908 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:12.917260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:23.125 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:23.134123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:23.133 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:23.142323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:33.353 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:33.362135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:33.365 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:33.374368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:43.586 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:43.592240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:43.593 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:43.602660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:53.813 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:53.823025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:28:53.826 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:28:53.835016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:04.046 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:04.056004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:04.059 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:04.068654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:12.070 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:12.080110 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T08:29:12.678 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:12.687281 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T08:29:12.678+0000 7fd6c74f8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T08:29:12.679 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:29:12.678+0000 7fd6c74f8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T08:29:12.679 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:29:12.678+0000 7fd6c74f8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T08:29:12.679 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T08:29:12.679 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:12.687306 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T08:29:12.678+0000 7fd6c74f8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T08:29:12.680 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:29:12.678+0000 7fd6c74f8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T08:29:12.680 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:29:12.678+0000 7fd6c74f8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T08:29:12.680 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T08:29:14.281 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:14.291043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:14.296 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:14.304447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:24.518 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:24.527546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:24.530 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:24.540099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:34.761 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:34.770352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:34.770 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:34.780961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:44.994 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:45.004181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:45.006 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:45.016834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:55.228 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:55.238335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:29:55.240 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:29:55.250878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:05.464 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:05.474763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:05.477 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:05.487413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:15.701 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:15.711219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:15.715 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:15.725866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:25.935 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:25.945546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:25.947 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:25.957617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:36.174 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:36.184419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:36.188 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:36.198577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:46.414 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:46.424399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:46.426 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:46.437200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:56.653 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:56.664132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:30:56.663 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:30:56.673763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:06.895 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:06.906065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:06.906 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:06.917473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:17.136 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:17.146888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:17.149 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:17.160165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:27.368 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:27.378649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:27.380 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:27.391244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:37.601 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:37.611539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:37.615 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:37.623964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:47.840 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:47.851304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:47.853 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:47.864300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:58.078 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:58.089106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:31:58.088 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:31:58.099928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:08.320 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:08.331049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:08.330 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:08.341563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:18.548 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:18.559245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:18.560 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:18.572193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:28.816 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:28.827219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:28.834 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:28.845594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:39.065 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:39.076255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:39.075 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:39.086910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:49.296 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:49.307696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:49.304 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:49.316347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:59.525 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:59.536577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:32:59.538 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:32:59.550429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:09.761 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:09.772709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:09.771 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:09.782542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:19.992 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:20.004100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:20.003 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:20.014631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:30.221 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:30.231805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:30.232 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:30.244268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:40.459 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:40.471484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:40.471 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:40.482816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:50.686 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:50.697873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:33:50.701 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:33:50.713655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:00.926 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:00.937230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:00.937 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:00.949511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:11.161 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:11.173483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:11.173 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:11.185657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:21.397 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:21.408684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:21.409 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:21.421313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:31.636 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:31.648505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:31.648 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:31.661150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:41.869 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:41.881488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:41.883 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:41.893684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:52.104 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:52.116047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:34:52.117 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:34:52.129895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:02.342 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:02.354312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:02.393 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:02.404560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:12.622 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:12.634962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:12.636 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:12.648803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:22.870 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:22.882282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:22.880 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:22.892645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:33.101 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:33.113922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:33.114 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:33.127870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:43.346 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:43.358870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:43.359 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:43.372077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:53.580 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:53.592220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:35:53.592 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:35:53.605191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:03.812 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:03.825109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:03.822 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:03.835679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:14.047 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:14.060067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:14.060 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:14.073282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:24.279 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:24.291297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:24.287 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:24.300462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:34.532 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:34.545214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:34.544 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:34.557824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:44.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:44.778351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:44.796 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:44.809979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:55.012 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:55.025362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:36:55.025 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:36:55.039044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:05.272 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:05.285389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:05.283 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:05.296903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:15.499 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:15.512867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:15.513 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:15.526292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:25.734 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:25.747510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:25.749 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:25.762607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:36.003 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:36.017393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:36.013 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:36.026462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:46.232 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:46.245437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:46.244 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:46.258169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:56.467 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:56.480443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:37:56.481 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:37:56.494435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:06.704 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:06.713609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:06.714 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:06.728257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:16.937 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:16.950357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:16.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:16.961603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:27.170 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:27.183418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:27.184 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:27.197905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:37.409 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:37.422335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:37.423 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:37.436832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:47.653 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:47.666770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:47.663 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:47.676949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:57.902 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:57.915773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:38:57.914 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:38:57.928730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:08.143 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:08.156915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:08.159 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:08.173641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:18.377 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:18.390469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:18.389 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:18.402923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:28.619 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:28.630770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:28.628 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:28.642172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:38.852 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:38.865908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:38.864 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:38.878465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:49.208 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:49.221290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:49.215 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:49.229276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:59.436 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:59.449744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:39:59.450 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:39:59.464421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:09.670 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:09.683853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:09.747 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:09.761530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:19.983 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:19.997543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:19.999 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:20.014126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:30.213 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:30.227502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:30.224 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:30.239185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:40.488 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:40.501473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:40.497 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:40.511117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:50.716 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:50.730132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:40:50.725 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:40:50.739498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:00.943 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:00.958028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:00.956 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:00.970523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:11.197 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:11.211324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:11.206 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:11.220155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:21.425 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:21.439163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:21.438 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:21.452253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:31.657 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:31.671441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:31.670 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:31.684699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:41.889 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:41.903521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:41.904 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:41.918771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:52.160 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:52.174407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:41:52.172 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:41:52.186540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:02.310 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:02.324688 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 08:42:02.315973428 +0000 UTC m=+7263.625774045 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T08:42:02.423 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:02.437195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:02.433 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:02.448395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:12.683 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:12.697518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:12.691 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:12.706001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:22.909 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:22.923882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:22.917 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:22.931844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:33.138 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:33.152713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:33.148 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:33.163748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:43.363 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:43.377907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:43.374 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:43.389545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:53.599 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:53.614189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:42:53.611 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:42:53.625940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:03.833 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:03.847430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:03.845 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:03.860549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:14.068 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:14.082717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:14.082 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:14.096959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:24.309 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:24.323374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:24.319 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:24.334385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:34.543 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:34.558378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:34.551 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:34.566328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:44.773 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:44.788260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:44.784 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:44.799423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:55.015 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:55.029697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:43:55.028 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:43:55.043815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:05.282 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:05.297361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:05.294 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:05.309620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:15.550 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:15.565715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:15.563 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:15.578639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:25.801 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:25.816889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:25.814 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:25.829894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:36.028 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:36.043759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:36.039 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:36.054590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:46.255 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:46.270735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:46.294 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:46.308939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:56.554 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:56.569664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:44:56.566 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:44:56.581795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:06.782 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:06.796873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:06.794 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:06.809452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:17.013 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:17.027983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:17.024 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:17.039892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:27.248 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:27.263562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:27.261 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:27.276788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:37.488 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:37.503423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:37.499 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:37.514956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:47.722 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:47.737392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:47.735 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:47.750908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:52.681 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:52.697201 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T08:45:53.319 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:53.334116 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T08:45:53.323+0000 7f0296f1f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T08:45:53.319 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:45:53.327+0000 7f0296f1f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T08:45:53.319 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:45:53.327+0000 7f0296f1f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T08:45:53.320 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T08:45:53.320 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:53.334143 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T08:45:53.323+0000 7f0296f1f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T08:45:53.321 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:45:53.327+0000 7f0296f1f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T08:45:53.321 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T08:45:53.327+0000 7f0296f1f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T08:45:53.321 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T08:45:57.957 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:57.972630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:45:57.969 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:45:57.984660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:08.196 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:08.212012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:08.207 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:08.221501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:18.426 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:18.441392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:18.438 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:18.454525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:28.658 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:28.674143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:28.672 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:28.688153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:38.892 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:38.907733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:38.904 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:38.920177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:49.126 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:49.141343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:49.139 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:49.154767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:59.364 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:59.380024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:46:59.377 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:46:59.393664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:09.602 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:09.617660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:09.614 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:09.630376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:19.838 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:19.853510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:19.846 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:19.862208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:30.069 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:30.084369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:30.084 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:30.100533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:40.304 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:40.320268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:40.316 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:40.332098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:50.540 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:50.556425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:47:50.557 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:47:50.573848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:00.783 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:00.798979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:00.796 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:00.811963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:11.020 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:11.035866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:11.033 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:11.048672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:21.259 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:21.275543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:21.271 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:21.287385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:31.514 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:31.530005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:31.525 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:31.541548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:41.754 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:41.770560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:41.824 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:41.840803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:52.045 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:52.061240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:48:52.056 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:48:52.072582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:02.307 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:02.323517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:02.335 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:02.351793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:12.554 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:12.569804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:12.640 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:12.583993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:22.786 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:22.802942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:22.799 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:22.815352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:33.022 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:33.038582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:33.033 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:33.049862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:43.255 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:43.271583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:43.264 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:43.280919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:53.489 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:53.505220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:49:53.500 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:49:53.516849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:03.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:03.745341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:03.741 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:03.757669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:13.987 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:14.003348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:14.002 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:14.014998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:24.218 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:24.234535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:24.231 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:24.248460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:34.453 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:34.470245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:34.464 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:34.480794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:44.681 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:44.697741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:44.693 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:44.709539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:54.910 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:54.927289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:50:54.925 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:50:54.942468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:05.148 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:05.164807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:05.157 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:05.174466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:15.383 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:15.399362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:15.395 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:15.411807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:25.625 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:25.641583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:25.634 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:25.651103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:35.859 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:35.876199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:35.868 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:35.885527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:46.099 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:46.115696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:46.112 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:46.128981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:56.337 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:56.354303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:51:56.355 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:51:56.372058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:06.574 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:06.591102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:06.584 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:06.600787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:16.823 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:16.839619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:16.837 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:16.853913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:27.074 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:27.090409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:27.088 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:27.104664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:37.318 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:37.334709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:37.331 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:37.347623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:47.581 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:47.598198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:47.589 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:47.606299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:57.846 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:57.863039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:52:57.872 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:52:57.888864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:08.119 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:08.135683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:08.132 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:08.149509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:18.404 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:18.420918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:18.417 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:18.433921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:28.636 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:28.652041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:28.648 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:28.664344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:38.867 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:38.882943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:38.879 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:38.895565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:49.102 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:49.117455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:49.113 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:49.129280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:59.336 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:59.351433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:53:59.348 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:53:59.364243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:09.570 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:09.585251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:09.580 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:09.595464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:19.809 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:19.823589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:19.822 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:19.836744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:30.044 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:30.058973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:30.058 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:30.072162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:40.277 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:40.291149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:40.290 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:40.304334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:50.516 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:50.529267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:54:50.526 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:54:50.540530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:00.754 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:00.760886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:00.760 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:00.773782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:10.978 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:10.991405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:10.991 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:11.004117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:21.211 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:21.223621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:21.223 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:21.236153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:31.443 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:31.455972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:31.456 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:31.469647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:41.681 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:41.693311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:41.693 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:41.705842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:51.917 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:51.928828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:55:51.929 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:55:51.941250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:02.151 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:02.162553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:02.163 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:02.175071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:12.390 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:12.401697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:12.401 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:12.412892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:22.631 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:22.642948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:22.647 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:22.658382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:32.870 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:32.880959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:32.881 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:32.892616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:43.107 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:43.118219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:43.120 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:43.131507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:53.346 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:53.356966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:56:53.396 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:56:53.407350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:03.640 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:03.650093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:03.654 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:03.664298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:13.921 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:13.930452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:13.957 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:13.967380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:24.184 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:24.193215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:24.195 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:24.205586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:34.415 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:34.424923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:34.427 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:34.436762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:44.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:44.675701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:44.680 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:44.688625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:54.897 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:54.906636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:57:54.907 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:57:54.916683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:05.130 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:05.138999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:05.148 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:05.151625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:15.374 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:15.383231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:15.387 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:15.395968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:25.613 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:25.621908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:25.625 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:25.633712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:35.849 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:35.858022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:35.859 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:35.867751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:46.076 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:46.084700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:46.089 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:46.097249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:56.321 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:56.328946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:58:56.334 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:58:56.342549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:06.563 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:06.570560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:06.573 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:06.581599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:16.793 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:16.800690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:16.817 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:16.820779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:17.079 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:17.087281 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 08:59:17.071586163 +0000 UTC m=+8298.381386811 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T08:59:27.053 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:27.059632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:27.064 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:27.070820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:37.279 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:37.286724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:37.291 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:37.299120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:47.522 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:47.528563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:47.533 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:47.539858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:57.754 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:57.760427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T08:59:57.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 08:59:57.772208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:07.986 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:07.992789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:07.999 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:08.006231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:18.247 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:18.253300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:18.259 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:18.265849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:28.480 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:28.486097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:28.492 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:28.498858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:38.720 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:38.722964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:38.735 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:38.735565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:48.954 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:48.959800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:48.999 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:49.005539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:59.216 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:59.221970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:00:59.230 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:00:59.236311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:09.489 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:09.495328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:09.500 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:09.505874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:19.716 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:19.721692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:19.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:19.735282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:30.022 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:30.026163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:30.034 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:30.038637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:40.258 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:40.263400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:40.269 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:40.274654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:50.506 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:50.510975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:01:50.517 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:01:50.522324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:00.762 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:00.766853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:00.778 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:00.781444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:10.991 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:10.995453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:11.001 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:11.006810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:21.226 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:21.230868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:21.239 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:21.243683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:31.456 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:31.460505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:31.470 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:31.475623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:33.336 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:33.340299 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T09:02:33.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:33.951520 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T09:02:33.933+0000 7f20977cd340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T09:02:33.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:02:33.933+0000 7f20977cd340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T09:02:33.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:02:33.933+0000 7f20977cd340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T09:02:33.949 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T09:02:33.949 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:33.951559 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T09:02:33.933+0000 7f20977cd340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T09:02:33.949 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:02:33.933+0000 7f20977cd340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T09:02:33.949 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:02:33.933+0000 7f20977cd340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T09:02:33.950 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T09:02:41.694 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:41.699040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:41.706 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:41.711126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:51.936 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:51.939936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:02:51.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:02:51.952832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:02.174 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:02.178283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:02.186 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:02.190359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:12.411 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:12.415104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:12.422 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:12.426201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:22.643 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:22.646606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:22.656 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:22.660269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:32.886 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:32.890355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:32.900 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:32.904413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:43.126 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:43.129126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:43.136 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:43.140494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:53.354 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:53.357177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:03:53.365 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:03:53.369196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:03.591 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:03.594335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:03.603 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:03.606505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:13.829 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:13.832405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:13.841 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:13.845122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:24.067 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:24.070774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:24.077 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:24.080723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:34.330 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:34.332970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:34.342 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:34.345235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:44.591 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:44.594490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:44.603 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:44.606677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:54.852 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:54.854258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:04:54.872 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:04:54.874929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:05.131 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:05.134383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:05.139 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:05.142943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:15.372 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:15.375293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:15.385 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:15.388184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:25.615 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:25.617924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:25.627 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:25.630499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:35.920 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:35.922792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:35.931 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:35.934336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:46.153 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:46.155001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:46.162 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:46.165307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:56.380 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:56.382690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:05:56.394 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:05:56.396462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:06.628 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:06.622057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:06.630 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:06.632992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:16.853 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:16.854987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:16.866 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:16.868524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:27.091 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:27.092978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:27.103 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:27.105657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:37.325 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:37.327500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:37.373 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:37.375696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:47.614 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:47.615842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:47.628 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:47.629508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:57.874 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:57.876424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:06:57.884 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:06:57.886434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:08.108 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:08.109695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:08.121 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:08.123078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:18.341 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:18.342751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:18.354 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:18.356540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:28.580 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:28.581989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:28.592 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:28.593729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:38.812 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:38.813199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:38.823 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:38.825222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:49.047 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:49.049097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:49.058 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:49.060102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:59.280 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:59.281278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:07:59.291 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:07:59.292391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:09.517 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:09.518639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:09.528 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:09.529949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:19.750 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:19.751179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:19.763 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:19.764003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:29.988 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:29.989196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:30.010 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:30.003439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:40.227 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:40.228136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:40.240 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:40.241407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:50.500 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:50.501022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:08:50.512 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:08:50.513169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:00.756 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:00.756653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:00.767 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:00.768471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:10.992 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:10.992008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:11.015 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:11.015974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:21.233 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:21.234389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:21.262 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:21.262934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:31.515 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:31.515928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:31.528 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:31.529180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:41.745 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:41.746079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:41.754 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:41.755365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:51.974 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:51.974637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:09:51.986 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:09:51.986953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:02.206 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:02.206230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:02.218 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:02.219218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:12.440 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:12.440323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:12.457 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:12.457664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:22.727 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:22.726922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:22.739 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:22.739790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:32.968 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:32.967853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:32.979 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:32.978290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:43.243 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:43.243045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:43.256 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:43.256903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:53.481 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:53.481387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:10:53.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:10:53.494575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:03.728 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:03.727994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:03.741 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:03.741522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:14.002 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:14.002477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:14.014 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:14.014335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:24.235 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:24.235100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:24.248 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:24.248205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:34.470 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:34.469801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:34.482 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:34.482056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:44.700 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:44.700006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:44.712 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:44.711894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:54.939 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:54.938743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:11:54.949 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:11:54.949839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:05.177 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:05.176543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:05.189 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:05.189300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:15.410 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:15.409651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:15.424 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:15.424191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:25.651 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:25.650080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:25.661 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:25.660782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:35.888 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:35.886713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:35.899 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:35.899009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:46.132 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:46.131371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:46.190 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:46.188882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:56.411 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:56.409784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:12:56.422 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:12:56.422467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:06.656 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:06.655278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:06.668 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:06.667339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:16.889 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:16.889011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:16.900 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:16.900010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:27.119 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:27.118942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:27.133 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:27.132591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:37.351 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:37.349679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:37.364 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:37.363175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:47.619 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:47.618039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:47.630 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:47.630141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:57.874 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:57.872707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:13:58.053 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:13:57.887365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:08.103 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:08.102726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:08.115 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:08.114524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:18.336 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:18.335285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:18.349 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:18.348686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:28.573 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:28.571843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:28.587 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:28.586781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:38.809 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:38.808440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:38.821 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:38.818977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:49.036 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:49.035292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:49.050 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:49.048980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:59.272 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:59.270736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:14:59.285 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:14:59.284240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:09.506 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:09.505077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:09.521 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:09.519203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:19.743 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:19.741429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:19.754 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:19.753692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:29.981 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:29.979870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:29.995 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:29.993800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:40.219 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:40.216079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:40.230 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:40.228612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:50.459 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:50.457628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:15:50.500 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:15:50.495516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:00.740 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:00.738729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:00.754 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:00.752867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:11.022 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:11.020702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:11.037 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:11.035852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:21.257 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:21.254833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:21.275 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:21.273362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:21.275 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:21.273387 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:31.495 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:31.493331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:31.505 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:31.504183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:31.738 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:31.737275 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 09:16:31.731405794 +0000 UTC m=+9333.041206392 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T09:16:41.725 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:41.723724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:41.736 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:41.735173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:51.996 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:51.994300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:16:52.024 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:16:52.022528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:02.243 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:02.241857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:02.256 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:02.254186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:12.472 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:12.470441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:12.483 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:12.481776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:22.730 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:22.728868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:22.745 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:22.743484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:33.028 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:33.026587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:33.038 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:33.036468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:43.253 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:43.251818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:43.266 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:43.264357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:53.482 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:53.480768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:17:53.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:17:53.492780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:03.716 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:03.714783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:03.782 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:03.780025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:14.049 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:14.047810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:14.060 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:14.058412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:24.282 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:24.280184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:24.296 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:24.293867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:34.517 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:34.515119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:34.530 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:34.529099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:44.753 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:44.751063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:44.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:44.763713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:54.991 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:54.988825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:18:55.004 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:18:55.002018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:05.231 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:05.228845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:05.245 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:05.243263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:13.962 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:13.960547 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T09:19:14.628 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:14.625751 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T09:19:14.614+0000 7ff40960a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T09:19:14.629 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:19:14.614+0000 7ff40960a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T09:19:14.629 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:19:14.614+0000 7ff40960a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T09:19:14.629 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T09:19:14.629 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:14.625796 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T09:19:14.614+0000 7ff40960a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T09:19:14.630 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:19:14.614+0000 7ff40960a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T09:19:14.630 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:19:14.614+0000 7ff40960a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T09:19:14.630 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T09:19:15.467 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:15.465501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:15.479 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:15.477593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:25.707 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:25.704864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:25.719 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:25.717752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:35.941 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:35.938708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:35.953 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:35.951195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:46.177 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:46.175416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:46.191 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:46.189114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:56.412 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:56.410426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:19:56.423 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:19:56.421862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:06.661 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:06.659405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:06.676 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:06.674482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:16.910 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:16.906681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:16.919 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:16.916856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:27.185 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:27.182346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:27.197 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:27.195372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:37.425 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:37.422331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:37.438 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:37.436126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:47.660 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:47.657347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:47.673 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:47.670930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:57.900 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:57.897483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:20:57.945 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:20:57.943543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:08.159 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:08.156905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:08.179 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:08.177525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:18.402 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:18.399976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:18.412 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:18.409791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:28.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:28.664890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:28.677 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:28.675644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:38.897 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:38.894621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:38.908 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:38.906701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:49.127 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:49.124648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:49.140 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:49.137955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:59.357 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:59.354924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:21:59.371 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:21:59.368662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:09.594 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:09.591936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:09.606 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:09.604234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:19.826 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:19.823667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:19.836 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:19.833859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:30.058 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:30.055417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:30.070 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:30.067598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:40.289 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:40.286653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:40.299 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:40.296688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:50.522 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:50.519615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:22:50.567 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:22:50.565509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:00.792 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:00.789732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:00.805 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:00.801489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:11.030 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:11.027256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:11.046 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:11.044030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:21.270 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:21.267627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:21.283 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:21.281302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:31.505 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:31.502936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:31.518 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:31.515349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:41.742 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:41.738987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:41.753 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:41.750901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:51.977 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:51.973688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:23:51.987 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:23:51.984910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:02.204 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:02.201370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:02.217 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:02.214668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:12.442 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:12.438761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:12.453 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:12.450558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:22.677 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:22.674324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:22.686 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:22.684013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:32.921 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:32.918125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:32.932 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:32.927711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:43.161 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:43.153974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:43.172 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:43.169517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:53.396 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:53.392873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:24:53.413 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:24:53.411069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:03.633 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:03.630804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:03.641 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:03.638595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:13.876 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:13.874037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:13.889 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:13.885776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:24.126 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:24.123450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:24.134 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:24.131924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:34.372 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:34.369857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:34.380 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:34.377382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:44.603 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:44.600711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:44.618 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:44.614945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:54.838 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:54.834728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:25:54.850 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:25:54.847511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:05.066 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:05.063568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:05.080 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:05.077207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:15.300 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:15.296886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:15.313 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:15.310538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:25.537 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:25.533642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:25.549 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:25.546776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:35.774 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:35.770714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:35.787 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:35.784604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:46.006 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:46.003642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:46.020 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:46.016789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:56.272 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:56.269107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:26:56.291 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:26:56.287191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:06.551 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:06.548039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:06.565 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:06.561834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:16.784 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:16.780645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:16.797 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:16.793363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:27.020 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:27.016589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:27.033 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:27.030027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:37.256 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:37.252663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:37.266 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:37.263535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:47.492 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:47.488954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:47.504 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:47.501838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:57.737 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:57.734724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:27:57.750 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:27:57.747373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:07.970 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:07.966959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:07.979 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:07.976551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:18.243 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:18.240472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:18.255 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:18.252335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:28.480 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:28.477080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:28.493 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:28.489933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:38.716 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:38.713696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:38.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:38.726892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:48.959 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:48.948307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:48.962 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:48.959854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:59.211 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:59.208602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:28:59.282 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:28:59.279116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:09.513 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:09.510695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:09.612 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:09.609781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:19.870 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:19.867884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:19.883 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:19.880783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:30.104 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:30.101986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:30.119 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:30.116559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:40.387 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:40.383784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:40.398 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:40.396354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:50.616 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:50.614083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:29:50.627 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:29:50.624938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:00.849 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:00.846367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:00.860 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:00.857650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:11.082 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:11.079174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:11.095 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:11.092280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:21.315 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:21.313197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:21.326 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:21.324132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:31.543 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:31.541092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:31.551 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:31.549282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:41.775 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:41.771861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:41.786 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:41.783631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:52.010 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:52.008027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:30:52.022 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:30:52.018218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:02.243 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:02.240667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:02.255 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:02.253342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:12.513 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:12.511492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:12.521 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:12.519874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:22.774 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:22.772043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:22.784 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:22.782483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:33.006 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:33.003661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:33.021 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:33.018487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:43.242 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:43.240069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:43.254 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:43.252736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:53.481 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:53.479009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:31:53.493 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:31:53.491077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:03.711 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:03.709327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:03.724 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:03.722083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:13.947 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:13.944939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:13.959 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:13.957408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:24.184 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:24.181810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:24.195 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:24.193871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:34.424 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:34.422228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:34.437 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:34.435261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:44.660 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:44.657808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:44.675 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:44.672825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:54.898 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:54.895857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:32:54.963 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:32:54.960901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:05.208 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:05.205668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:05.219 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:05.217816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:15.467 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:15.465557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:15.510 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:15.508782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:25.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:25.727542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:25.737 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:25.735902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:35.976 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:35.974140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:35.987 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:35.985994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:46.237 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:46.235607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:46.248 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:46.246343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:46.568 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:46.566013 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 09:33:46.554508924 +0000 UTC m=+10367.864309565 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T09:33:56.467 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:56.465315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:33:56.480 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:33:56.478879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:06.702 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:06.700462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:06.712 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:06.711012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:16.939 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:16.937148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:16.952 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:16.950817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:27.176 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:27.174057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:27.188 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:27.186148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:37.406 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:37.404538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:37.418 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:37.416803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:47.491 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:47.489729 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 09:34:47.430231676 +0000 UTC m=+10428.740032354 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T09:34:47.637 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:47.635685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:47.652 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:47.650587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:57.881 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:57.879912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:34:57.911 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:34:57.905471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:08.125 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:08.123423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:08.138 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:08.136607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:18.365 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:18.359970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:18.375 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:18.373229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:28.604 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:28.602207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:28.614 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:28.612861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:38.839 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:38.837131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:38.855 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:38.853717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:49.077 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:49.075610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:49.091 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:49.089494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:54.647 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:54.645142 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T09:35:55.339 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:55.336999 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T09:35:55.328+0000 7fbe902dd340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T09:35:55.339 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:35:55.328+0000 7fbe902dd340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T09:35:55.340 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:35:55.328+0000 7fbe902dd340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T09:35:55.340 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T09:35:55.340 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:55.337027 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T09:35:55.328+0000 7fbe902dd340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T09:35:55.340 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:35:55.328+0000 7fbe902dd340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T09:35:55.341 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:35:55.328+0000 7fbe902dd340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T09:35:55.341 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T09:35:59.308 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:59.306192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:35:59.319 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:35:59.318058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:09.545 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:09.543154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:09.557 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:09.556026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:19.784 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:19.782251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:19.797 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:19.795205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:30.018 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:30.016854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:30.029 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:30.028175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:40.254 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:40.252895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:40.268 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:40.266947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:50.488 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:50.486522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:36:50.502 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:36:50.500577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:00.730 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:00.727845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:00.742 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:00.740459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:10.962 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:10.960479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:10.976 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:10.974822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:21.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:21.192773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:21.206 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:21.204650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:31.424 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:31.422463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:31.438 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:31.435340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:41.652 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:41.650509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:41.664 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:41.662598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:51.882 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:51.880890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:37:51.960 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:37:51.958791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:02.182 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:02.180285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:02.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:02.193231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:12.417 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:12.415105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:12.426 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:12.424831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:22.650 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:22.649174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:22.664 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:22.661496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:32.884 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:32.882514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:32.936 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:32.934562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:43.157 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:43.155856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:43.169 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:43.167648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:53.388 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:53.386498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:38:53.400 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:38:53.399136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:03.622 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:03.620590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:03.635 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:03.633616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:13.863 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:13.862030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:13.877 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:13.875942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:24.102 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:24.101258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:24.111 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:24.110137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:34.339 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:34.336733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:34.350 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:34.348780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:44.572 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:44.571135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:44.588 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:44.587127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:54.812 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:54.811121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:39:54.824 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:39:54.822598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:05.055 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:05.053869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:05.068 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:05.066719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:15.295 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:15.291833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:15.304 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:15.303550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:25.533 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:25.531742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:25.546 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:25.545236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:35.776 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:35.774148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:35.793 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:35.791592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:46.014 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:46.012512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:46.029 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:46.028087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:56.256 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:56.255052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:40:56.270 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:40:56.268892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:06.491 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:06.488930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:06.502 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:06.501214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:16.729 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:16.727507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:16.740 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:16.739066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:26.965 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:26.962400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:26.980 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:26.978849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:37.216 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:37.214586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:37.231 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:37.229763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:47.538 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:47.536950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:47.550 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:47.549036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:57.772 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:57.770740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:41:57.783 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:41:57.782217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:08.004 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:08.003130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:08.017 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:08.015647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:18.238 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:18.237093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:18.247 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:18.245793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:28.473 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:28.471154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:28.488 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:28.486821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:38.712 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:38.711006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:38.723 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:38.721902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:48.971 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:48.969711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:48.980 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:48.978801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:59.202 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:59.200655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:42:59.214 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:42:59.212928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:09.435 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:09.433398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:09.447 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:09.446028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:19.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:19.666062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:19.681 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:19.679742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:29.904 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:29.902721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:29.916 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:29.915525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:40.144 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:40.142404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:40.157 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:40.156215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:50.382 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:50.380495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:43:50.393 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:43:50.392043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:00.620 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:00.618570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:00.632 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:00.631486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:10.856 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:10.855134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:10.865 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:10.864157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:21.088 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:21.086518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:21.100 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:21.099074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:31.353 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:31.351379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:31.364 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:31.362967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:41.592 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:41.590702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:41.601 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:41.600729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:51.832 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:51.830428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:44:51.911 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:44:51.910149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:02.137 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:02.135527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:02.147 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:02.146050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:12.375 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:12.373391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:12.442 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:12.440645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:22.674 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:22.673176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:22.686 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:22.684554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:32.905 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:32.903953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:32.915 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:32.914044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:43.153 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:43.151769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:43.165 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:43.163697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:53.443 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:53.441988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:45:53.453 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:45:53.452224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:03.675 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:03.673365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:03.687 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:03.685692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:13.912 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:13.910940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:13.928 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:13.922417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:24.142 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:24.141022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:24.153 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:24.152584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:34.380 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:34.378835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:34.392 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:34.390563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:44.606 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:44.604663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:44.619 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:44.617840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:54.869 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:54.863332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:46:54.876 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:46:54.875092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:05.098 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:05.096351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:05.109 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:05.108001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:15.391 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:15.389495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:15.404 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:15.402985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:25.628 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:25.626307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:25.640 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:25.639405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:35.862 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:35.860974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:35.875 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:35.874453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:46.096 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:46.094534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:46.109 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:46.108130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:56.357 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:56.355450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:47:56.369 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:47:56.367529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:06.601 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:06.599515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:06.617 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:06.616287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:16.840 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:16.838179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:16.850 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:16.849070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:27.084 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:27.082107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:27.095 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:27.094144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:37.316 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:37.314440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:37.328 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:37.326800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:47.554 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:47.552991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:47.567 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:47.565829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:57.790 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:57.788698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:48:57.802 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:48:57.800847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:08.030 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:08.028405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:08.039 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:08.038437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:18.304 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:18.303152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:18.315 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:18.314445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:28.531 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:28.529604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:28.543 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:28.541572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:38.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:38.763685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:38.778 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:38.776168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:48.998 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:48.996115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:49.006 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:49.004628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:59.246 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:59.244697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:49:59.256 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:49:59.254362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:09.522 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:09.520870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:09.546 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:09.544596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:19.773 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:19.770925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:19.785 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:19.783586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:30.002 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:30.000321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:30.015 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:30.013755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:40.235 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:40.233759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:40.248 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:40.246976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:50.473 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:50.471257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:50:50.485 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:50:50.484050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:00.712 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:00.710733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:00.723 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:00.722338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:10.953 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:10.951810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:10.965 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:10.964001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:21.187 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:21.185731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:21.198 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:21.197180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:31.423 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:31.421666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:31.438 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:31.435250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:41.654 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:41.652742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:41.665 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:41.664417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:51.886 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:51.884271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:51:51.895 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:51:51.894498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:02.116 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:02.114376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:02.127 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:02.126332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:12.346 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:12.344482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:12.356 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:12.355190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:22.575 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:22.573460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:22.588 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:22.587434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:32.811 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:32.809349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:32.822 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:32.821488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:35.346 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:35.344620 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T09:52:35.971 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:35.969022 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T09:52:35.957+0000 7f9572482340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T09:52:35.971 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:52:35.957+0000 7f9572482340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T09:52:35.971 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:52:35.957+0000 7f9572482340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T09:52:35.972 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T09:52:35.972 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:35.969075 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T09:52:35.957+0000 7f9572482340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T09:52:35.972 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:52:35.957+0000 7f9572482340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T09:52:35.972 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T09:52:35.957+0000 7f9572482340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T09:52:35.973 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T09:52:43.047 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:43.045664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:43.060 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:43.058840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:53.280 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:53.279053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:52:53.295 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:52:53.293596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:03.515 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:03.514004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:03.527 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:03.526114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:03.589 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:03.588284 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 09:53:03.545378643 +0000 UTC m=+11524.855179250 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T09:53:13.750 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:13.748971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:13.764 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:13.762997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:23.988 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:23.986876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:24.001 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:23.999790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:34.218 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:34.216731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:34.227 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:34.226079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:44.471 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:44.469700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:44.484 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:44.482567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:54.703 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:54.701428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:53:54.714 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:53:54.713180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:04.964 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:04.962455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:04.979 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:04.977899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:15.198 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:15.196500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:15.259 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:15.257305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:25.563 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:25.561494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:25.576 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:25.574731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:35.796 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:35.794299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:35.809 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:35.808093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:46.028 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:46.027143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:46.036 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:46.035482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:56.409 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:56.407393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:54:56.421 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:54:56.420505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:06.645 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:06.643524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:06.658 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:06.656497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:16.888 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:16.886376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:16.929 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:16.928240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:27.152 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:27.150303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:27.161 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:27.159736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:37.406 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:37.404165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:37.415 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:37.414324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:47.634 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:47.632446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:47.643 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:47.642036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:58.061 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:58.059554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:55:58.076 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:55:58.074594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:08.292 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:08.290659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:08.305 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:08.303439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:18.525 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:18.523601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:18.536 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:18.535413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:28.766 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:28.764838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:28.780 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:28.778691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:39.006 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:39.004282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:39.018 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:39.017325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:49.251 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:49.249114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:49.258 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:49.257424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:59.492 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:59.490178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:56:59.506 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:56:59.505238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:09.732 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:09.730268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:09.762 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:09.759849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:19.994 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:19.991928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:20.009 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:20.007280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:30.228 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:30.226742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:30.239 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:30.237964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:40.490 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:40.488681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:40.502 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:40.500813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:50.720 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:50.718465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:57:50.730 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:57:50.728875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:00.966 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:00.964430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:01.029 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:01.027277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:11.248 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:11.245919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:11.263 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:11.261673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:21.492 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:21.489830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:21.511 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:21.509318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:31.731 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:31.729526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:31.744 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:31.742719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:41.969 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:41.967265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:41.985 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:41.983138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:52.208 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:52.206739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:58:52.223 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:58:52.222088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:02.446 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:02.444210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:02.459 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:02.457008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:12.681 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:12.679310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:12.692 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:12.690924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:22.914 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:22.912708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:22.926 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:22.924486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:33.152 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:33.150442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:33.164 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:33.162810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:43.401 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:43.399790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:43.409 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:43.408112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:53.673 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:53.671350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T09:59:53.684 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 09:59:53.683120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:03.943 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:03.941555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:03.952 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:03.950968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:14.211 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:14.209838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:14.224 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:14.222460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:24.445 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:24.443996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:24.453 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:24.451713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:34.676 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:34.674133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:34.688 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:34.686952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:44.913 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:44.911325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:44.924 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:44.922629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:55.146 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:55.144423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:00:55.160 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:00:55.158680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:05.385 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:05.382913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:05.404 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:05.402594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:15.654 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:15.652548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:15.667 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:15.665741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:25.894 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:25.892300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:25.907 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:25.906069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:36.141 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:36.139043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:36.152 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:36.151120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:46.412 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:46.410350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:46.427 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:46.425474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:56.663 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:56.660829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:01:56.675 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:01:56.673492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:06.891 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:06.888765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:06.899 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:06.898270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:17.133 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:17.131443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:17.145 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:17.143835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:27.366 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:27.364451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:27.377 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:27.376457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:37.594 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:37.592857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:37.606 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:37.605284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:47.831 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:47.830034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:47.843 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:47.842241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:58.099 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:58.097433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:02:58.108 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:02:58.107036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:08.335 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:08.333090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:08.348 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:08.346671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:18.572 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:18.570507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:18.583 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:18.582178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:28.806 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:28.804419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:28.814 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:28.812641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:39.041 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:39.039916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:39.056 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:39.055162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:49.273 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:49.272044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:49.287 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:49.285979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:59.518 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:59.516925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:03:59.530 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:03:59.529201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:09.756 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:09.754733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:09.768 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:09.767145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:20.005 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:20.003716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:20.016 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:20.015026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:30.239 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:30.238038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:30.251 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:30.249582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:40.467 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:40.466354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:40.477 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:40.476453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:50.705 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:50.704200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:04:50.717 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:04:50.715870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:00.953 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:00.951257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:00.965 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:00.964342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:11.184 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:11.183560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:11.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:11.192902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:21.474 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:21.473102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:21.486 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:21.484941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:31.707 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:31.705814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:31.718 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:31.716830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:41.938 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:41.936735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:41.947 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:41.945882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:52.167 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:52.166372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:05:52.180 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:05:52.179016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:02.398 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:02.397097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:02.411 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:02.409897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:12.638 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:12.637417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:12.651 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:12.650800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:22.903 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:22.901673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:22.912 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:22.911947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:33.166 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:33.165612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:33.176 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:33.175801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:43.395 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:43.394128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:43.406 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:43.405546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:53.626 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:53.625536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:06:53.640 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:06:53.639116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:03.860 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:03.859625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:03.875 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:03.874106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:14.102 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:14.101301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:14.113 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:14.112033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:24.337 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:24.336048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:24.349 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:24.347704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:34.564 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:34.563406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:34.577 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:34.576601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:44.805 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:44.803703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:44.818 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:44.817744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:55.040 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:55.035691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:07:55.048 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:07:55.047851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:05.273 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:05.272249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:05.284 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:05.283288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:15.508 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:15.507569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:15.524 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:15.523007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:25.750 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:25.749342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:25.763 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:25.762237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:35.984 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:35.982885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:35.996 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:35.995383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:46.237 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:46.236229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:46.250 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:46.248775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:56.494 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:56.493417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:08:56.507 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:08:56.506299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:06.733 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:06.732575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:06.747 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:06.746530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:15.985 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:15.984703 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T10:09:16.703 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:16.701609 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T10:09:16.691+0000 7f136c21a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T10:09:16.703 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T10:09:16.691+0000 7f136c21a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T10:09:16.704 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T10:09:16.691+0000 7f136c21a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T10:09:16.704 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T10:09:16.704 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:16.701630 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T10:09:16.691+0000 7f136c21a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T10:09:16.704 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T10:09:16.691+0000 7f136c21a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T10:09:16.705 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T10:09:16.691+0000 7f136c21a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T10:09:16.705 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T10:09:16.960 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:16.959980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:16.969 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:16.968690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:17.352 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:17.351465 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 10:09:17.345737438 +0000 UTC m=+12498.655538095 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T10:09:27.212 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:27.211003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:27.254 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:27.253035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:37.475 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:37.474752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:37.488 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:37.487668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:47.708 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:47.707066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:47.717 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:47.716935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:57.936 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:57.935095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:09:57.955 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:09:57.954535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:08.174 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:08.172595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:08.185 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:08.184677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:18.231 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:18.229865 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 10:10:18.201385687 +0000 UTC m=+12559.511186324 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T10:10:18.426 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:18.425642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:18.437 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:18.436429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:28.653 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:28.651165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:28.664 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:28.663707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:38.890 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:38.883675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:39.692 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:38.894645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:49.117 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:49.116534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:49.128 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:49.128080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:59.348 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:59.347515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:10:59.361 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:10:59.360588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:09.578 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:09.577793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:09.590 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:09.589541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:19.812 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:19.811016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:19.825 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:19.824395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:30.046 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:30.045458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:30.058 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:30.058366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:40.281 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:40.280412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:40.294 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:40.293595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:50.515 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:50.514175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:11:50.525 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:11:50.525353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:00.747 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:00.746034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:00.758 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:00.757967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:10.979 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:10.978639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:10.995 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:10.994858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:21.217 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:21.216658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:21.231 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:21.230244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:31.451 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:31.449674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:31.463 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:31.462537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:41.690 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:41.689070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:41.703 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:41.702482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:51.928 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:51.927813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:12:51.943 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:12:51.942631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:02.167 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:02.166146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:02.179 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:02.179150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:12.405 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:12.401708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:12.449 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:12.448816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:22.673 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:22.672580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:22.686 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:22.685408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:32.964 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:32.963519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:32.979 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:32.978667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:43.202 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:43.201760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:43.210 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:43.209692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:53.429 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:53.428293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:13:53.438 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:13:53.438260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:03.683 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:03.682796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:03.697 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:03.697208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:13.927 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:13.926744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:13.942 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:13.941644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:24.187 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:24.186202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:24.199 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:24.199151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:34.419 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:34.418380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:34.430 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:34.429343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:44.652 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:44.651780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:44.665 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:44.665012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:54.887 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:54.887109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:14:54.902 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:14:54.901341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:05.127 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:05.126116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:05.138 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:05.137962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:15.376 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:15.375142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:15.389 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:15.388691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:25.609 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:25.608044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:25.622 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:25.621388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:35.839 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:35.838135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:35.851 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:35.851219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:46.073 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:46.072671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:46.121 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:46.121476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:56.452 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:56.451512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:15:56.463 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:15:56.462847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:06.691 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:06.690052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:06.701 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:06.701168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:16.925 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:16.924131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:16.937 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:16.936775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:27.165 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:27.165325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:27.176 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:27.176195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:27.177 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:27.176228 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:37.408 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:37.408379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:37.419 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:37.419307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:47.647 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:47.646200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:47.660 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:47.659917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:57.886 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:57.886113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:16:57.898 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:16:57.898197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:08.371 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:08.370316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:08.426 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:08.425737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:18.669 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:18.669192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:18.700 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:18.699796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:29.038 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:29.038063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:29.052 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:29.051703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:39.322 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:39.321983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:39.335 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:39.334714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:49.558 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:49.557719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:49.569 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:49.569293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:59.789 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:59.788292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:17:59.941 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:17:59.941315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:10.157 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:10.157493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:10.169 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:10.169471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:20.389 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:20.389203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:20.400 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:20.400318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:30.621 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:30.620771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:30.632 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:30.632097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:40.856 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:40.855922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:40.866 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:40.866482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:51.090 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:51.089800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:18:51.105 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:18:51.104715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:01.327 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:01.326322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:01.338 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:01.338168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:11.556 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:11.555309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:11.566 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:11.566489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:21.792 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:21.791002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:21.800 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:21.800238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:32.033 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:32.032419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:32.048 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:32.046760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:42.277 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:42.276695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:42.289 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:42.288731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:52.510 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:52.509884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:19:52.520 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:19:52.519806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:02.736 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:02.735836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:02.788 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:02.787883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:13.032 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:13.031757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:13.043 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:13.043069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:23.262 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:23.262262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:23.275 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:23.275073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:33.503 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:33.500431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:33.516 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:33.513926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:43.734 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:43.733997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:43.746 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:43.745979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:53.972 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:53.972165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:20:53.985 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:20:53.985543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:04.220 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:04.214914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:04.225 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:04.224233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:14.443 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:14.442728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:14.456 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:14.456064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:24.680 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:24.679123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:24.697 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:24.696957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:34.916 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:34.915963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:34.928 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:34.928127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:45.148 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:45.147747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:45.161 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:45.160536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:55.378 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:55.377911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:21:55.431 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:21:55.431339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:05.651 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:05.650791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:05.666 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:05.665672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:15.888 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:15.886691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:15.900 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:15.900259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:26.123 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:26.122938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:26.135 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:26.135155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:36.360 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:36.359141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:36.372 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:36.371840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:46.595 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:46.593992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:46.606 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:46.605760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:56.828 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:56.827647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:22:56.840 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:22:56.840581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:07.064 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:07.063739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:07.074 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:07.074441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:17.298 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:17.297689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:17.315 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:17.314788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:27.539 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:27.538020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:27.583 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:27.583490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:37.806 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:37.805547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:37.817 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:37.816626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:48.039 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:48.039099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:48.050 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:48.049401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:58.272 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:58.272157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:23:58.285 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:23:58.285605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:08.515 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:08.515139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:08.523 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:08.523082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:18.745 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:18.744752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:18.757 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:18.757289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:28.985 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:28.984836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:28.997 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:28.997569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:39.223 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:39.222705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:39.238 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:39.238322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:49.467 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:49.466728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:49.480 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:49.479442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:59.704 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:59.703686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:24:59.717 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:24:59.717153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:09.948 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:09.947414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:09.958 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:09.958552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:20.181 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:20.180674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:20.192 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:20.192451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:30.419 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:30.418964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:30.431 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:30.430723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:40.653 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:40.653088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:40.670 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:40.669644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:50.901 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:50.900640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:50.919 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:50.918441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:25:56.712 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:56.711665 I | op-mon: parsing mon endpoints: a=10.103.147.119:6789 2022-03-02T10:25:57.309 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:57.308186 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T10:25:57.300+0000 7ff768858340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T10:25:57.309 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T10:25:57.300+0000 7ff768858340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T10:25:57.310 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T10:25:57.300+0000 7ff768858340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T10:25:57.310 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T10:25:57.310 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:25:57.308213 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-02T10:25:57.300+0000 7ff768858340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-02T10:25:57.311 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T10:25:57.300+0000 7ff768858340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-02T10:25:57.311 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02T10:25:57.300+0000 7ff768858340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-02T10:25:57.311 INFO:tasks.rook.operator.smithi136.stdout:couldn't init storage provider. : exit status 5 2022-03-02T10:26:01.182 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:01.181537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:01.194 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:01.193520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:11.411 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:11.410221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:11.420 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:11.420395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:21.643 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:21.642319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:21.653 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:21.653001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:31.872 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:31.872012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:31.882 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:31.882462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:32.071 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:32.071543 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-02 10:26:32.065289215 +0000 UTC m=+13533.375089860 LastTransitionTime:2022-03-02 06:43:01 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-02T10:26:42.105 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:42.105090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:42.123 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:42.123207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:52.346 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:52.345750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:26:52.359 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:26:52.356964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:02.580 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:02.580337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:02.594 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:02.594100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:12.816 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:12.816251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:12.829 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:12.828747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:23.055 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:23.054476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:23.065 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:23.065212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:33.286 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:33.285548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:33.298 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:33.297613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:43.524 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:43.523836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:43.536 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:43.536029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:53.761 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:53.760515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:27:53.778 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:27:53.777771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:04.002 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:04.001707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:04.024 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:04.017766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:14.253 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:14.252245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:14.263 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:14.261586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:24.515 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:24.514367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:24.530 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:24.530805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:34.753 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:34.752460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:34.766 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:34.765885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:44.993 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:44.992259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:45.005 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:45.005195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:55.228 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:55.227833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:28:55.241 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:28:55.241102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:05.473 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:05.472701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:05.485 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:05.485541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:15.707 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:15.706683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:15.720 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:15.719628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:25.937 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:25.936628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:25.949 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:25.948869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:36.171 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:36.170998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:36.220 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:36.219584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:46.438 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:46.437554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:46.485 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:46.484877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:56.755 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:56.754246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:29:56.767 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:29:56.766509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:06.995 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:06.994762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:07.006 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:07.006455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:17.225 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:17.224383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:17.291 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:17.290794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:27.509 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:27.508257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:27.521 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:27.521273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:37.744 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:37.743378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:37.757 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:37.756424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:47.975 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:47.974453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:47.988 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:47.987587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:58.206 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:58.205856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:30:58.217 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:30:58.217025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:08.440 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:08.439870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:08.452 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:08.451642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:18.675 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:18.674630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:18.687 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:18.687474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:28.919 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:28.913837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:28.926 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:28.925691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:39.152 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:39.151674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:39.165 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:39.164743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:49.384 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:49.383932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:49.397 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:49.396766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:59.625 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:59.625099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:31:59.635 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:31:59.634447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:09.859 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:09.858362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:09.872 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:09.871584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:20.093 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:20.092490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:20.107 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:20.107446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:30.430 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:30.429501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:30.471 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:30.470756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:40.735 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:40.734005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:40.747 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:40.745445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:50.980 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:50.980161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:32:50.988 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:32:50.988260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:01.212 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:01.211214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:01.224 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:01.224220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:11.451 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:11.450730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:11.493 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:11.488482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:21.712 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:21.711585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:21.723 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:21.722828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:31.944 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:31.943900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:31.956 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:31.955606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:42.211 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:42.210479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:42.222 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:42.222603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:52.443 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:52.442378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:33:52.455 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:33:52.453941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:02.671 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:02.671289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:02.682 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:02.681789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:12.903 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:12.902296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:12.915 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:12.915541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:23.162 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:23.162107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:23.172 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:23.172457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:33.412 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:33.411317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:33.424 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:33.423579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:43.649 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:43.645988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:43.660 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:43.658968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:53.877 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:53.876218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:34:53.889 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:34:53.888723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:04.109 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:04.109220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:04.120 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:04.120413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:14.341 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:14.340840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:14.352 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:14.351845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:24.572 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:24.571657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:24.584 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:24.584467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:34.810 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:34.809684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:34.824 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:34.823471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:45.045 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:45.044808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:45.056 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:45.055893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:55.281 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:55.280316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:35:55.292 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:35:55.292360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:05.518 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:05.517742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:05.532 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:05.531362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:15.765 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:15.765123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:15.776 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:15.776314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:25.998 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:25.997930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:26.007 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:26.007132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:36.234 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:36.233548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:36.246 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:36.246021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:46.475 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:46.474801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:46.488 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:46.487907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:56.726 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:56.725710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:36:56.738 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:36:56.738548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:06.964 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:06.963710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:06.977 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:06.976496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:17.198 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:17.197444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:17.209 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:17.208636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:27.428 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:27.427666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:27.442 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:27.441354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:37.659 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:37.656604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:37.672 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:37.671816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:47.899 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:47.898981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:47.912 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:47.912244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:58.131 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:58.130710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:37:58.141 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:37:58.141068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:08.363 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:08.362484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:08.554 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:08.375010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:18.599 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:18.596584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:18.607 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:18.607142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:28.844 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:28.843672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:28.856 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:28.855641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:39.075 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:39.074208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:39.086 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:39.085852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:49.307 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:49.305322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:49.316 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:49.315362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:59.535 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:59.534766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:38:59.549 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:38:59.548592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:09.782 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:09.781583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:09.796 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:09.796081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:20.022 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:20.021369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:20.096 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:20.094811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:30.314 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:30.313318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:30.327 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:30.326770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:40.549 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:40.548003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:40.619 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:40.560438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:50.806 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:50.806100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:39:50.819 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:39:50.819046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:01.039 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:01.038690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:01.057 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:01.057433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:11.280 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:11.278885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:11.347 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:11.291144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:21.515 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:21.513824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:21.526 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:21.525850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:31.748 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:31.747175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:31.760 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:31.759503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:41.983 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:41.982567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:42.053 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:41.994968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:52.216 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:52.215289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:40:52.229 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:40:52.228782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:41:02.457 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:41:02.456758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T10:41:02.469 INFO:tasks.rook.operator.smithi136.stdout:2022-03-02 10:41:02.468467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-02T13:02:49.258 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-02T13:02:49.405 DEBUG:teuthology.task.console_log:Killing console logger for smithi136 2022-03-02T13:02:49.407 DEBUG:teuthology.exit:Finished running handlers