2022-03-10T18:38:24.630 INFO:root:teuthology version: 1.1.0-ff5db916 2022-03-10T18:38:24.632 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_ozeneva@teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/host rook/master} --verbose --name ozeneva-2022-03-10_12:34:28-rados-wip-omri-tracing-compiled-distro-basic-smithi --archive /home/teuthworker/archive/ozeneva-2022-03-10_12:34:28-rados-wip-omri-tracing-compiled-distro-basic-smithi/6730121 -- /home/teuthworker/archive/ozeneva-2022-03-10_12:34:28-rados-wip-omri-tracing-compiled-distro-basic-smithi/6730121/orig.config.yaml 2022-03-10T18:38:24.676 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-10T18:38:24.744 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ozeneva-2022-03-10_12:34:28-rados-wip-omri-tracing-compiled-distro-basic-smithi/6730121 branch: wip-omri-tracing-compiled description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/host rook/master} email: null first_in_suite: false job_id: '6730121' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ozeneva-2022-03-10_12:34:28-rados-wip-omri-tracing-compiled-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-omri-tracing-compiled ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 78e1eb94d97cd01be975676a7cefdde0c403b210 spec: mon: allowMultiplePerNode: false network: provider: host ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 78e1eb94d97cd01be975676a7cefdde0c403b210 kernel: hwe: true kubeadm: version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-omri-tracing-compiled sha1: 78e1eb94d97cd01be975676a7cefdde0c403b210 owner: scheduled_ozeneva@teuthology priority: 100 repo: git://git.ceph.com/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 78e1eb94d97cd01be975676a7cefdde0c403b210 sleep_before_teardown: 0 suite: rados suite_branch: wip-omri-tracing-compiled suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 78e1eb94d97cd01be975676a7cefdde0c403b210 targets: smithi093.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCc/RbqhhEmTudl3/DxqAwqHvcRMoxWcxLwcUIUDsKS8Ii89wOPslibyit31DCIkC3GUBNlpJTyvcmTZPSKBLk0HlWBu/yEEFEC7VSN13GFMdWWETpESC7lPfd+h7GB8zKQB43CH9Xwkn0UwWk5HTuZWznTLnq+z7gqn9HXw7tarkdoZrQ9sZoJ2b7/+Mq9oNDQygVXgt1pKGRWz+xxk/3YvroGKb4l9UeMObDwp2t6aaEda+N99EDxBjgWwN984uXVmmVVP58MHrV9Q/w6jMsbvXgw4dwKd33097nMdlXwYaGQJArAMnaeLao20u9jtKtP9PoBVu1BwFdHLKS8LR0bIqBjGQxxN00dTiRP2Cr69qqu/PglqtC8/K1Upa6D5QxszEybtzY1bqRgTw57yaxPIhnP5FYrtDCw//5iBr8vJ43CJnL1brC34mK8J/MmmSA9QPcPzsNqXX7KOMG24D/08jYnQBX0d9QqWcVBsBJ27cSuvzwVcZG7NaQKD8cjQH8= smithi111.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnmi7Bbfw6rbIHeyRKLcdRGh9qrCtof17lrucbnyvlD605OQMepAO4pn0uJvd+ITX3QgVapBF0/5n33niPuhx6Sh+ov8ZZCYkJP2WTSUiTStYOUho08BmukFP5z5IU1GRKshgnW5xrg+iU4hWuIwA9RHq+5W4zJ/NViTGyAIteiynhNyaIkV1LWNCF9M5tmJwneFQCR47b7YJ0KluX2F9ODycdvc+5hEgfkf3KPrLryn1FnUb0k242lFmoqGKIGL1YwmbW5SsJ3xHQ9BiU6jsagQp0DU2ZNm4tnojN7YekTmtXkRrpAIIF/HWg/wJpBd49AgMqCTfzU7uPtLF7NlJVpMbG/nDKPBWKX8Owt57o7/ayQM3fFkTXeUUiSIYY2LdeMiDh4jkl7Mm9P/+INZ0Hi3cKKqVTAJKUvldJAs8rg4sCLMMRonjWnJbp5f4FZ6xgQQTSao6qOpFqgFulk+uva5D6ci0SOUl+CLdRyGsprznOb0piR/pbtK0eAvQyjh8= smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnHb9qvAPdjov7zfd3TsdhNRVKHosOfDOTDoSNrg0y4o7xvl+Ko0wy6Zvsx/vK2NCCV9LUjtm2OhVZiwn6y65is8NyRSY8uJmy+siND+NyyYPqx5U5SjiIfNqSDF5OFFHmkzd8owiYac8NxwthS4TVTyi6nAspVZFHV0nPPaCZLRVvsYm/FmozlXOG/CWprXlubGLPqO1l7CRBmt0beoZQ59oODWt7WT0Qva2RtRa4TZURKIKtUuxS8DPz+X64lOgtaRJp75rgD3lvshJgUVC3a1GnF0pRbDJUfmEjnEKep9fcMQrrhJ5goEwM/23J7cAWINRXefA+Os93c7+2xV//ORnbBjaM3GQJM2+zT8E8m/oOcIo5aK3Qc0FkMfojpuJwoQKNb8DHBTI6pQaciaF5FywTdZxqoymfnMLspqTI4ln9D1iB8/0lUcGqUsKfa+568LT75ZeeF7pxknASZJjTkMU696050Sm82uFnUGI49Rqba4yt6N3nwL5qEucd2Pk= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: ff5db9169566776b2f41faf671316ca58a318ea6 timestamp: 2022-03-10_12:34:28 tube: smithi user: ozeneva verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-10T18:38:24.745 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa; will attempt to use it 2022-03-10T18:38:24.746 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks 2022-03-10T18:38:24.746 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-10T18:38:24.751 INFO:teuthology.task.internal:Checking packages... 2022-03-10T18:38:24.773 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '78e1eb94d97cd01be975676a7cefdde0c403b210' 2022-03-10T18:38:24.773 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-10T18:38:24.774 INFO:teuthology.packaging:ref: None 2022-03-10T18:38:24.774 INFO:teuthology.packaging:tag: None 2022-03-10T18:38:24.774 INFO:teuthology.packaging:branch: wip-omri-tracing-compiled 2022-03-10T18:38:24.775 INFO:teuthology.packaging:sha1: 78e1eb94d97cd01be975676a7cefdde0c403b210 2022-03-10T18:38:24.775 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-omri-tracing-compiled 2022-03-10T18:38:24.954 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11084-g78e1eb94-1focal 2022-03-10T18:38:24.955 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-10T18:38:24.972 INFO:teuthology.task.internal:no buildpackages task found 2022-03-10T18:38:24.972 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-10T18:38:25.012 INFO:teuthology.task.internal:Saving configuration 2022-03-10T18:38:25.025 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-10T18:38:25.039 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-10T18:38:25.064 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi093.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-10_12:34:28-rados-wip-omri-tracing-compiled-distro-basic-smithi/6730121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 18:32:34.607958', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICBrOaUX7UmP2Nml5Y2ThBpDJe+ZfbfcIV9iXKBVcp1u'} 2022-03-10T18:38:25.086 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-10_12:34:28-rados-wip-omri-tracing-compiled-distro-basic-smithi/6730121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 18:32:34.606662', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDjwKfMcR0FC2SHk8fVHWa51iCakqt3xA1Ugupl6UbHo'} 2022-03-10T18:38:25.110 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-10_12:34:28-rados-wip-omri-tracing-compiled-distro-basic-smithi/6730121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 18:32:34.604384', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMpF1lv+hnSDNAAtFgzYnDFxCxW0i8NpIDGDiwQogA8Z'} 2022-03-10T18:38:25.110 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-10T18:38:25.122 INFO:teuthology.task.internal:roles: ubuntu@smithi093.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-10T18:38:25.122 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-10T18:38:25.123 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-10T18:38:25.123 INFO:teuthology.run_tasks:Running task console_log... 2022-03-10T18:38:25.283 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5f62aad950>, signals=[15]) 2022-03-10T18:38:25.285 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-10T18:38:25.296 INFO:teuthology.task.internal:Opening connections... 2022-03-10T18:38:25.296 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi093.front.sepia.ceph.com 2022-03-10T18:38:25.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:38:25.351 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2022-03-10T18:38:25.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:38:25.402 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2022-03-10T18:38:25.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:38:25.452 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-10T18:38:25.463 DEBUG:teuthology.orchestra.run.smithi093:> uname -m 2022-03-10T18:38:25.471 INFO:teuthology.orchestra.run.smithi093.stdout:x86_64 2022-03-10T18:38:25.472 DEBUG:teuthology.orchestra.run.smithi093:> cat /etc/os-release 2022-03-10T18:38:25.521 INFO:teuthology.orchestra.run.smithi093.stdout:NAME="Ubuntu" 2022-03-10T18:38:25.522 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T18:38:25.522 INFO:teuthology.orchestra.run.smithi093.stdout:ID=ubuntu 2022-03-10T18:38:25.522 INFO:teuthology.orchestra.run.smithi093.stdout:ID_LIKE=debian 2022-03-10T18:38:25.523 INFO:teuthology.orchestra.run.smithi093.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T18:38:25.523 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION_ID="20.04" 2022-03-10T18:38:25.523 INFO:teuthology.orchestra.run.smithi093.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T18:38:25.523 INFO:teuthology.orchestra.run.smithi093.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T18:38:25.524 INFO:teuthology.orchestra.run.smithi093.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T18:38:25.524 INFO:teuthology.orchestra.run.smithi093.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T18:38:25.524 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION_CODENAME=focal 2022-03-10T18:38:25.525 INFO:teuthology.orchestra.run.smithi093.stdout:UBUNTU_CODENAME=focal 2022-03-10T18:38:25.525 INFO:teuthology.lock.ops:Updating smithi093.front.sepia.ceph.com on lock server 2022-03-10T18:38:25.545 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2022-03-10T18:38:25.552 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2022-03-10T18:38:25.552 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2022-03-10T18:38:25.600 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="Ubuntu" 2022-03-10T18:38:25.601 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T18:38:25.601 INFO:teuthology.orchestra.run.smithi111.stdout:ID=ubuntu 2022-03-10T18:38:25.601 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE=debian 2022-03-10T18:38:25.601 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T18:38:25.602 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="20.04" 2022-03-10T18:38:25.602 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T18:38:25.602 INFO:teuthology.orchestra.run.smithi111.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T18:38:25.602 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T18:38:25.603 INFO:teuthology.orchestra.run.smithi111.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T18:38:25.603 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_CODENAME=focal 2022-03-10T18:38:25.603 INFO:teuthology.orchestra.run.smithi111.stdout:UBUNTU_CODENAME=focal 2022-03-10T18:38:25.604 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2022-03-10T18:38:25.623 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2022-03-10T18:38:25.630 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2022-03-10T18:38:25.631 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2022-03-10T18:38:25.679 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="Ubuntu" 2022-03-10T18:38:25.680 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T18:38:25.680 INFO:teuthology.orchestra.run.smithi161.stdout:ID=ubuntu 2022-03-10T18:38:25.680 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE=debian 2022-03-10T18:38:25.680 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T18:38:25.681 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="20.04" 2022-03-10T18:38:25.681 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T18:38:25.681 INFO:teuthology.orchestra.run.smithi161.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T18:38:25.681 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T18:38:25.681 INFO:teuthology.orchestra.run.smithi161.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T18:38:25.682 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_CODENAME=focal 2022-03-10T18:38:25.682 INFO:teuthology.orchestra.run.smithi161.stdout:UBUNTU_CODENAME=focal 2022-03-10T18:38:25.682 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2022-03-10T18:38:25.700 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-10T18:38:25.714 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-10T18:38:25.746 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-10T18:38:25.746 DEBUG:teuthology.orchestra.run.smithi093:> test '!' -e /home/ubuntu/cephtest 2022-03-10T18:38:25.749 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2022-03-10T18:38:25.751 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2022-03-10T18:38:25.760 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-10T18:38:25.782 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-10T18:38:25.783 DEBUG:teuthology.orchestra.run.smithi093:> test -z $(ls -A /var/lib/ceph) 2022-03-10T18:38:25.785 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2022-03-10T18:38:25.788 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2022-03-10T18:38:25.809 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-10T18:38:25.892 INFO:teuthology.run_tasks:Running task kernel... 2022-03-10T18:38:25.913 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-10T18:38:25.914 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-10T18:38:25.915 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-10T18:38:25.915 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-10T18:38:25.916 DEBUG:teuthology.orchestra.run.smithi093:> uname -r 2022-03-10T18:38:25.916 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2022-03-10T18:38:25.917 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2022-03-10T18:38:25.924 INFO:teuthology.orchestra.run.smithi111.stdout:5.4.0-97-generic 2022-03-10T18:38:25.924 INFO:teuthology.task.kernel:Running kernel on smithi111: 5.4.0-97-generic 2022-03-10T18:38:25.925 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get clean 2022-03-10T18:38:25.927 INFO:teuthology.orchestra.run.smithi161.stdout:5.4.0-97-generic 2022-03-10T18:38:25.927 INFO:teuthology.orchestra.run.smithi093.stdout:5.4.0-97-generic 2022-03-10T18:38:25.928 INFO:teuthology.task.kernel:Running kernel on smithi161: 5.4.0-97-generic 2022-03-10T18:38:25.928 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-get clean 2022-03-10T18:38:25.929 INFO:teuthology.task.kernel:Running kernel on smithi093: 5.4.0-97-generic 2022-03-10T18:38:25.929 DEBUG:teuthology.orchestra.run.smithi093:> sudo apt-get clean 2022-03-10T18:38:25.994 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2022-03-10T18:38:26.027 DEBUG:teuthology.orchestra.run.smithi093:> sudo apt-get update 2022-03-10T18:38:26.036 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-get update 2022-03-10T18:38:26.162 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T18:38:26.192 INFO:teuthology.orchestra.run.smithi093.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T18:38:26.268 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:38:26.301 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:38:26.349 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:38:26.350 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T18:38:26.359 INFO:teuthology.orchestra.run.smithi161.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T18:38:26.383 INFO:teuthology.orchestra.run.smithi093.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T18:38:26.384 INFO:teuthology.orchestra.run.smithi161.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T18:38:26.468 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,316 kB] 2022-03-10T18:38:26.476 INFO:teuthology.orchestra.run.smithi093.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,316 kB] 2022-03-10T18:38:26.577 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [401 kB] 2022-03-10T18:38:26.589 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [230 kB] 2022-03-10T18:38:26.590 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,764 B] 2022-03-10T18:38:26.590 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T18:38:26.590 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T18:38:26.593 INFO:teuthology.orchestra.run.smithi093.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [401 kB] 2022-03-10T18:38:26.596 INFO:teuthology.orchestra.run.smithi111.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T18:38:26.598 INFO:teuthology.orchestra.run.smithi111.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T18:38:26.598 INFO:teuthology.orchestra.run.smithi111.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [691 kB] 2022-03-10T18:38:26.600 INFO:teuthology.orchestra.run.smithi093.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [230 kB] 2022-03-10T18:38:26.604 INFO:teuthology.orchestra.run.smithi093.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,764 B] 2022-03-10T18:38:26.605 INFO:teuthology.orchestra.run.smithi093.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T18:38:26.606 INFO:teuthology.orchestra.run.smithi093.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T18:38:26.612 INFO:teuthology.orchestra.run.smithi111.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-10T18:38:26.617 INFO:teuthology.orchestra.run.smithi111.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [120 kB] 2022-03-10T18:38:26.618 INFO:teuthology.orchestra.run.smithi111.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-10T18:38:26.619 INFO:teuthology.orchestra.run.smithi111.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T18:38:26.620 INFO:teuthology.orchestra.run.smithi111.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T18:38:26.625 INFO:teuthology.orchestra.run.smithi093.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T18:38:26.626 INFO:teuthology.orchestra.run.smithi093.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T18:38:26.627 INFO:teuthology.orchestra.run.smithi093.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [691 kB] 2022-03-10T18:38:26.632 INFO:teuthology.orchestra.run.smithi093.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-10T18:38:26.637 INFO:teuthology.orchestra.run.smithi093.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [120 kB] 2022-03-10T18:38:26.638 INFO:teuthology.orchestra.run.smithi093.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-10T18:38:26.639 INFO:teuthology.orchestra.run.smithi111.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T18:38:26.639 INFO:teuthology.orchestra.run.smithi111.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T18:38:26.648 INFO:teuthology.orchestra.run.smithi093.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T18:38:26.649 INFO:teuthology.orchestra.run.smithi093.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T18:38:26.649 INFO:teuthology.orchestra.run.smithi093.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T18:38:26.649 INFO:teuthology.orchestra.run.smithi093.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T18:38:26.692 INFO:teuthology.orchestra.run.smithi111.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T18:38:26.723 INFO:teuthology.orchestra.run.smithi161.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T18:38:26.725 INFO:teuthology.orchestra.run.smithi093.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T18:38:26.818 INFO:teuthology.orchestra.run.smithi111.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [616 kB] 2022-03-10T18:38:26.857 INFO:teuthology.orchestra.run.smithi093.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [616 kB] 2022-03-10T18:38:26.862 INFO:teuthology.orchestra.run.smithi161.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [616 kB] 2022-03-10T18:38:26.998 INFO:teuthology.orchestra.run.smithi161.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [401 kB] 2022-03-10T18:38:27.072 INFO:teuthology.orchestra.run.smithi111.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,641 kB] 2022-03-10T18:38:27.109 INFO:teuthology.orchestra.run.smithi093.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,641 kB] 2022-03-10T18:38:27.147 INFO:teuthology.orchestra.run.smithi161.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,641 kB] 2022-03-10T18:38:27.215 INFO:teuthology.orchestra.run.smithi111.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T18:38:27.234 INFO:teuthology.orchestra.run.smithi111.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T18:38:27.234 INFO:teuthology.orchestra.run.smithi111.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T18:38:27.236 INFO:teuthology.orchestra.run.smithi111.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T18:38:27.253 INFO:teuthology.orchestra.run.smithi093.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T18:38:27.261 INFO:teuthology.orchestra.run.smithi111.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T18:38:27.266 INFO:teuthology.orchestra.run.smithi111.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T18:38:27.267 INFO:teuthology.orchestra.run.smithi111.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-10T18:38:27.267 INFO:teuthology.orchestra.run.smithi093.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T18:38:27.268 INFO:teuthology.orchestra.run.smithi093.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T18:38:27.273 INFO:teuthology.orchestra.run.smithi161.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,316 kB] 2022-03-10T18:38:27.273 INFO:teuthology.orchestra.run.smithi093.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T18:38:27.291 INFO:teuthology.orchestra.run.smithi111.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-10T18:38:27.291 INFO:teuthology.orchestra.run.smithi161.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T18:38:27.303 INFO:teuthology.orchestra.run.smithi161.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T18:38:27.304 INFO:teuthology.orchestra.run.smithi161.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T18:38:27.305 INFO:teuthology.orchestra.run.smithi093.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T18:38:27.310 INFO:teuthology.orchestra.run.smithi093.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T18:38:27.311 INFO:teuthology.orchestra.run.smithi093.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-10T18:38:27.318 INFO:teuthology.orchestra.run.smithi111.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T18:38:27.322 INFO:teuthology.orchestra.run.smithi111.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-10T18:38:27.335 INFO:teuthology.orchestra.run.smithi093.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-10T18:38:27.338 INFO:teuthology.orchestra.run.smithi161.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T18:38:27.339 INFO:teuthology.orchestra.run.smithi161.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T18:38:27.344 INFO:teuthology.orchestra.run.smithi161.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T18:38:27.344 INFO:teuthology.orchestra.run.smithi161.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-10T18:38:27.349 INFO:teuthology.orchestra.run.smithi111.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T18:38:27.349 INFO:teuthology.orchestra.run.smithi111.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T18:38:27.350 INFO:teuthology.orchestra.run.smithi111.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T18:38:27.350 INFO:teuthology.orchestra.run.smithi111.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T18:38:27.351 INFO:teuthology.orchestra.run.smithi111.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T18:38:27.360 INFO:teuthology.orchestra.run.smithi093.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T18:38:27.363 INFO:teuthology.orchestra.run.smithi093.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-10T18:38:27.368 INFO:teuthology.orchestra.run.smithi161.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-10T18:38:27.380 INFO:teuthology.orchestra.run.smithi111.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T18:38:27.381 INFO:teuthology.orchestra.run.smithi111.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T18:38:27.393 INFO:teuthology.orchestra.run.smithi093.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T18:38:27.393 INFO:teuthology.orchestra.run.smithi093.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T18:38:27.394 INFO:teuthology.orchestra.run.smithi093.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T18:38:27.394 INFO:teuthology.orchestra.run.smithi093.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T18:38:27.395 INFO:teuthology.orchestra.run.smithi093.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T18:38:27.395 INFO:teuthology.orchestra.run.smithi161.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T18:38:27.397 INFO:teuthology.orchestra.run.smithi161.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-10T18:38:27.419 INFO:teuthology.orchestra.run.smithi093.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T18:38:27.420 INFO:teuthology.orchestra.run.smithi093.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T18:38:27.426 INFO:teuthology.orchestra.run.smithi161.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [230 kB] 2022-03-10T18:38:27.427 INFO:teuthology.orchestra.run.smithi161.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T18:38:27.427 INFO:teuthology.orchestra.run.smithi161.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T18:38:27.428 INFO:teuthology.orchestra.run.smithi161.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T18:38:27.429 INFO:teuthology.orchestra.run.smithi161.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T18:38:27.429 INFO:teuthology.orchestra.run.smithi161.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T18:38:27.451 INFO:teuthology.orchestra.run.smithi161.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,764 B] 2022-03-10T18:38:27.451 INFO:teuthology.orchestra.run.smithi161.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T18:38:27.458 INFO:teuthology.orchestra.run.smithi161.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T18:38:27.458 INFO:teuthology.orchestra.run.smithi161.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T18:38:27.462 INFO:teuthology.orchestra.run.smithi111.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T18:38:27.463 INFO:teuthology.orchestra.run.smithi111.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T18:38:27.463 INFO:teuthology.orchestra.run.smithi111.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T18:38:27.464 INFO:teuthology.orchestra.run.smithi111.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T18:38:27.496 INFO:teuthology.orchestra.run.smithi161.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T18:38:27.496 INFO:teuthology.orchestra.run.smithi161.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T18:38:27.500 INFO:teuthology.orchestra.run.smithi161.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T18:38:27.500 INFO:teuthology.orchestra.run.smithi161.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [691 kB] 2022-03-10T18:38:27.501 INFO:teuthology.orchestra.run.smithi093.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T18:38:27.501 INFO:teuthology.orchestra.run.smithi093.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T18:38:27.502 INFO:teuthology.orchestra.run.smithi093.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T18:38:27.502 INFO:teuthology.orchestra.run.smithi093.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T18:38:27.526 INFO:teuthology.orchestra.run.smithi161.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-10T18:38:27.540 INFO:teuthology.orchestra.run.smithi161.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T18:38:27.541 INFO:teuthology.orchestra.run.smithi161.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T18:38:27.541 INFO:teuthology.orchestra.run.smithi161.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T18:38:27.542 INFO:teuthology.orchestra.run.smithi161.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [120 kB] 2022-03-10T18:38:27.542 INFO:teuthology.orchestra.run.smithi161.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T18:38:27.544 INFO:teuthology.orchestra.run.smithi161.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-10T18:38:27.594 INFO:teuthology.orchestra.run.smithi161.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T18:38:27.595 INFO:teuthology.orchestra.run.smithi161.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T18:38:27.595 INFO:teuthology.orchestra.run.smithi161.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T18:38:27.596 INFO:teuthology.orchestra.run.smithi161.stdout:Get:43 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T18:38:29.891 INFO:teuthology.orchestra.run.smithi093.stdout:Fetched 10.2 MB in 2s (6,019 kB/s) 2022-03-10T18:38:29.934 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 10.2 MB in 2s (5,979 kB/s) 2022-03-10T18:38:30.469 INFO:teuthology.orchestra.run.smithi161.stdout:Fetched 10.2 MB in 2s (4,671 kB/s) 2022-03-10T18:38:31.061 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2022-03-10T18:38:31.086 DEBUG:teuthology.orchestra.run.smithi093:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T18:38:31.103 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-03-10T18:38:31.127 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T18:38:31.153 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2022-03-10T18:38:31.199 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-03-10T18:38:31.327 INFO:teuthology.orchestra.run.smithi093.stdout:Building dependency tree... 2022-03-10T18:38:31.328 INFO:teuthology.orchestra.run.smithi093.stdout:Reading state information... 2022-03-10T18:38:31.374 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-03-10T18:38:31.374 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-03-10T18:38:31.474 INFO:teuthology.orchestra.run.smithi093.stdout:The following additional packages will be installed: 2022-03-10T18:38:31.474 INFO:teuthology.orchestra.run.smithi093.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T18:38:31.474 INFO:teuthology.orchestra.run.smithi093.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T18:38:31.475 INFO:teuthology.orchestra.run.smithi093.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T18:38:31.475 INFO:teuthology.orchestra.run.smithi093.stdout:Suggested packages: 2022-03-10T18:38:31.476 INFO:teuthology.orchestra.run.smithi093.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T18:38:31.476 INFO:teuthology.orchestra.run.smithi093.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T18:38:31.520 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2022-03-10T18:38:31.521 INFO:teuthology.orchestra.run.smithi111.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T18:38:31.521 INFO:teuthology.orchestra.run.smithi111.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T18:38:31.521 INFO:teuthology.orchestra.run.smithi111.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T18:38:31.522 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2022-03-10T18:38:31.522 INFO:teuthology.orchestra.run.smithi111.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T18:38:31.522 INFO:teuthology.orchestra.run.smithi111.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T18:38:31.523 INFO:teuthology.orchestra.run.smithi093.stdout:The following NEW packages will be installed: 2022-03-10T18:38:31.524 INFO:teuthology.orchestra.run.smithi093.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T18:38:31.525 INFO:teuthology.orchestra.run.smithi093.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T18:38:31.525 INFO:teuthology.orchestra.run.smithi093.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T18:38:31.525 INFO:teuthology.orchestra.run.smithi093.stdout: thermald upower usbmuxd 2022-03-10T18:38:31.577 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2022-03-10T18:38:31.578 INFO:teuthology.orchestra.run.smithi111.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T18:38:31.578 INFO:teuthology.orchestra.run.smithi111.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T18:38:31.578 INFO:teuthology.orchestra.run.smithi111.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T18:38:31.579 INFO:teuthology.orchestra.run.smithi111.stdout: thermald upower usbmuxd 2022-03-10T18:38:31.669 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2022-03-10T18:38:31.692 DEBUG:teuthology.orchestra.run.smithi161:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T18:38:31.715 INFO:teuthology.orchestra.run.smithi093.stdout:0 upgraded, 13 newly installed, 0 to remove and 96 not upgraded. 2022-03-10T18:38:31.715 INFO:teuthology.orchestra.run.smithi093.stdout:Need to get 80.1 MB of archives. 2022-03-10T18:38:31.716 INFO:teuthology.orchestra.run.smithi093.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T18:38:31.716 INFO:teuthology.orchestra.run.smithi093.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T18:38:31.758 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2022-03-10T18:38:31.759 INFO:teuthology.orchestra.run.smithi111.stdout:0 upgraded, 13 newly installed, 0 to remove and 96 not upgraded. 2022-03-10T18:38:31.759 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 80.1 MB of archives. 2022-03-10T18:38:31.760 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T18:38:31.760 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T18:38:31.930 INFO:teuthology.orchestra.run.smithi161.stdout:Building dependency tree... 2022-03-10T18:38:31.931 INFO:teuthology.orchestra.run.smithi161.stdout:Reading state information... 2022-03-10T18:38:32.018 INFO:teuthology.orchestra.run.smithi093.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T18:38:32.020 INFO:teuthology.orchestra.run.smithi093.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T18:38:32.028 INFO:teuthology.orchestra.run.smithi093.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T18:38:32.029 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T18:38:32.043 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T18:38:32.044 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T18:38:32.071 INFO:teuthology.orchestra.run.smithi161.stdout:The following additional packages will be installed: 2022-03-10T18:38:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T18:38:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T18:38:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T18:38:32.073 INFO:teuthology.orchestra.run.smithi161.stdout:Suggested packages: 2022-03-10T18:38:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T18:38:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T18:38:32.115 INFO:teuthology.orchestra.run.smithi161.stdout:The following NEW packages will be installed: 2022-03-10T18:38:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T18:38:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T18:38:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T18:38:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: thermald upower usbmuxd 2022-03-10T18:38:32.120 INFO:teuthology.orchestra.run.smithi093.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T18:38:32.121 INFO:teuthology.orchestra.run.smithi093.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T18:38:32.130 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T18:38:32.131 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T18:38:32.297 INFO:teuthology.orchestra.run.smithi161.stdout:0 upgraded, 13 newly installed, 0 to remove and 96 not upgraded. 2022-03-10T18:38:32.297 INFO:teuthology.orchestra.run.smithi161.stdout:Need to get 80.1 MB of archives. 2022-03-10T18:38:32.297 INFO:teuthology.orchestra.run.smithi161.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T18:38:32.298 INFO:teuthology.orchestra.run.smithi161.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T18:38:32.583 INFO:teuthology.orchestra.run.smithi161.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T18:38:32.606 INFO:teuthology.orchestra.run.smithi161.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T18:38:32.607 INFO:teuthology.orchestra.run.smithi161.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T18:38:32.695 INFO:teuthology.orchestra.run.smithi161.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T18:38:32.709 INFO:teuthology.orchestra.run.smithi161.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T18:38:33.119 INFO:teuthology.orchestra.run.smithi093.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T18:38:33.251 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T18:38:33.534 INFO:teuthology.orchestra.run.smithi093.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T18:38:33.568 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T18:38:33.722 INFO:teuthology.orchestra.run.smithi161.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T18:38:34.233 INFO:teuthology.orchestra.run.smithi161.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T18:38:35.757 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T18:38:35.758 INFO:teuthology.orchestra.run.smithi111.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T18:38:35.759 INFO:teuthology.orchestra.run.smithi111.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T18:38:35.764 INFO:teuthology.orchestra.run.smithi111.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T18:38:35.766 INFO:teuthology.orchestra.run.smithi111.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi093.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi093.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi093.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi093.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi093.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T18:38:36.175 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 80.1 MB in 4s (19.1 MB/s) 2022-03-10T18:38:36.314 INFO:teuthology.orchestra.run.smithi161.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T18:38:36.315 INFO:teuthology.orchestra.run.smithi161.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T18:38:36.315 INFO:teuthology.orchestra.run.smithi161.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T18:38:36.321 INFO:teuthology.orchestra.run.smithi161.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T18:38:36.323 INFO:teuthology.orchestra.run.smithi161.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T18:38:36.379 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T18:38:36.396 INFO:teuthology.orchestra.run.smithi093.stdout:Fetched 80.1 MB in 4s (17.9 MB/s) 2022-03-10T18:38:36.614 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T18:38:36.721 INFO:teuthology.orchestra.run.smithi161.stdout:Fetched 80.1 MB in 4s (19.1 MB/s) 2022-03-10T18:38:36.915 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T18:38:37.866 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T18:38:37.871 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T18:38:37.991 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T18:38:38.053 INFO:teuthology.orchestra.run.smithi093.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T18:38:38.058 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T18:38:38.202 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T18:38:38.352 INFO:teuthology.orchestra.run.smithi161.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T18:38:38.356 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T18:38:38.386 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T18:38:38.399 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T18:38:38.436 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T18:38:38.470 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T18:38:38.530 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T18:38:38.543 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T18:38:38.597 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T18:38:38.764 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T18:38:38.778 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T18:38:38.822 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T18:38:38.883 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T18:38:38.883 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T18:38:38.894 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T18:38:38.897 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T18:38:38.931 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T18:38:38.941 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T18:38:39.151 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T18:38:39.163 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T18:38:39.200 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T18:38:39.225 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T18:38:39.241 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T18:38:39.300 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T18:38:39.319 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T18:38:39.334 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T18:38:39.377 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T18:38:39.537 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T18:38:39.551 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T18:38:39.595 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T18:38:39.612 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T18:38:39.626 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T18:38:39.670 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T18:38:39.755 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T18:38:39.770 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T18:38:39.814 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T18:38:39.931 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T18:38:39.946 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T18:38:39.990 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:40.006 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T18:38:40.021 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T18:38:40.064 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T18:38:40.125 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T18:38:40.139 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T18:38:40.174 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:40.375 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T18:38:40.387 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T18:38:40.446 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:42.266 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T18:38:42.279 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T18:38:42.372 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:42.408 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T18:38:42.425 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T18:38:42.523 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:42.650 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T18:38:42.665 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T18:38:42.772 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:42.869 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T18:38:42.910 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:43.028 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T18:38:43.040 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T18:38:43.078 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:43.263 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T18:38:43.752 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T18:38:43.752 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:49.027 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T18:38:49.042 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T18:38:49.085 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T18:38:49.087 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T18:38:49.102 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T18:38:49.145 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T18:38:49.258 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T18:38:49.274 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T18:38:49.313 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T18:38:49.414 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T18:38:49.430 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T18:38:49.472 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T18:38:49.480 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T18:38:49.495 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T18:38:49.538 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T18:38:49.656 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T18:38:49.670 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T18:38:49.714 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T18:38:49.842 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package thermald. 2022-03-10T18:38:49.856 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T18:38:49.891 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package thermald. 2022-03-10T18:38:49.900 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T18:38:49.904 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T18:38:49.941 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T18:38:50.068 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package thermald. 2022-03-10T18:38:50.081 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T18:38:50.109 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T18:38:50.449 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package upower. 2022-03-10T18:38:50.463 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T18:38:50.491 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package upower. 2022-03-10T18:38:50.504 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T18:38:50.517 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T18:38:50.559 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T18:38:50.659 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package upower. 2022-03-10T18:38:50.672 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T18:38:50.719 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T18:38:50.915 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T18:38:50.933 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T18:38:50.974 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T18:38:51.009 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T18:38:51.023 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T18:38:51.066 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T18:38:51.134 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T18:38:51.149 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T18:38:51.200 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T18:38:51.264 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T18:38:51.387 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T18:38:51.394 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T18:38:51.483 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T18:38:51.504 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T18:38:51.512 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T18:38:51.588 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T18:38:51.630 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T18:38:51.639 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T18:38:51.733 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T18:38:51.756 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T18:38:51.765 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T18:38:51.857 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T18:38:51.900 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T18:38:51.907 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T18:38:51.975 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T18:38:52.009 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T18:38:52.035 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T18:38:52.101 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T18:38:52.135 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T18:38:52.227 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T18:38:52.482 INFO:teuthology.orchestra.run.smithi093.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T18:38:52.534 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T18:38:52.592 INFO:teuthology.orchestra.run.smithi111.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T18:38:52.636 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T18:38:52.663 INFO:teuthology.orchestra.run.smithi161.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T18:38:52.712 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T18:38:52.716 INFO:teuthology.orchestra.run.smithi093.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T18:38:52.722 INFO:teuthology.orchestra.run.smithi093.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T18:38:52.722 INFO:teuthology.orchestra.run.smithi093.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T18:38:52.820 INFO:teuthology.orchestra.run.smithi111.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T18:38:52.829 INFO:teuthology.orchestra.run.smithi111.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T18:38:52.830 INFO:teuthology.orchestra.run.smithi111.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T18:38:52.890 INFO:teuthology.orchestra.run.smithi161.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T18:38:52.896 INFO:teuthology.orchestra.run.smithi161.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T18:38:52.896 INFO:teuthology.orchestra.run.smithi161.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T18:38:53.201 INFO:teuthology.orchestra.run.smithi093.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T18:38:53.286 INFO:teuthology.orchestra.run.smithi111.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T18:38:53.379 INFO:teuthology.orchestra.run.smithi161.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T18:38:53.442 INFO:teuthology.orchestra.run.smithi093.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T18:38:53.481 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T18:38:53.525 INFO:teuthology.orchestra.run.smithi111.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T18:38:53.566 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T18:38:53.600 INFO:teuthology.orchestra.run.smithi161.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T18:38:53.642 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T18:38:53.780 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T18:38:53.781 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T18:38:53.848 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T18:38:53.848 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T18:38:53.957 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T18:38:53.957 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T18:38:54.276 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:54.337 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:54.438 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:55.519 INFO:teuthology.orchestra.run.smithi093.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T18:38:55.520 INFO:teuthology.orchestra.run.smithi093.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T18:38:55.560 INFO:teuthology.orchestra.run.smithi111.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T18:38:55.561 INFO:teuthology.orchestra.run.smithi111.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T18:38:55.678 INFO:teuthology.orchestra.run.smithi161.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T18:38:55.678 INFO:teuthology.orchestra.run.smithi161.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T18:38:55.679 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:55.743 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:55.873 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:57.091 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:57.192 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:57.355 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:38:58.409 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T18:38:58.493 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T18:38:58.536 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T18:38:58.618 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T18:38:58.680 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T18:38:58.807 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T18:39:02.866 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T18:39:03.163 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T18:39:03.400 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T18:39:03.584 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T18:39:03.693 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:39:03.882 INFO:teuthology.orchestra.run.smithi161.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T18:39:03.882 INFO:teuthology.orchestra.run.smithi161.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T18:39:03.915 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T18:39:04.006 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:39:04.093 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T18:39:04.191 INFO:teuthology.orchestra.run.smithi093.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T18:39:04.191 INFO:teuthology.orchestra.run.smithi093.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T18:39:04.210 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T18:39:04.395 INFO:teuthology.orchestra.run.smithi111.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T18:39:04.396 INFO:teuthology.orchestra.run.smithi111.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T18:39:27.272 INFO:teuthology.orchestra.run.smithi161.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T18:39:27.273 INFO:teuthology.orchestra.run.smithi161.stdout:Sourcing file `/etc/default/grub' 2022-03-10T18:39:27.285 INFO:teuthology.orchestra.run.smithi161.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T18:39:27.309 INFO:teuthology.orchestra.run.smithi161.stdout:Generating grub configuration file ... 2022-03-10T18:39:27.597 INFO:teuthology.orchestra.run.smithi161.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T18:39:27.629 INFO:teuthology.orchestra.run.smithi161.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T18:39:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T18:39:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:Sourcing file `/etc/default/grub' 2022-03-10T18:39:27.779 INFO:teuthology.orchestra.run.smithi111.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T18:39:27.805 INFO:teuthology.orchestra.run.smithi111.stdout:Generating grub configuration file ... 2022-03-10T18:39:27.824 INFO:teuthology.orchestra.run.smithi093.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T18:39:27.825 INFO:teuthology.orchestra.run.smithi093.stdout:Sourcing file `/etc/default/grub' 2022-03-10T18:39:27.836 INFO:teuthology.orchestra.run.smithi093.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T18:39:27.862 INFO:teuthology.orchestra.run.smithi093.stdout:Generating grub configuration file ... 2022-03-10T18:39:28.037 INFO:teuthology.orchestra.run.smithi161.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T18:39:28.048 INFO:teuthology.orchestra.run.smithi161.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T18:39:28.112 INFO:teuthology.orchestra.run.smithi111.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T18:39:28.147 INFO:teuthology.orchestra.run.smithi111.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T18:39:28.178 INFO:teuthology.orchestra.run.smithi093.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T18:39:28.206 INFO:teuthology.orchestra.run.smithi093.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T18:39:28.525 INFO:teuthology.orchestra.run.smithi111.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T18:39:28.536 INFO:teuthology.orchestra.run.smithi111.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T18:39:28.581 INFO:teuthology.orchestra.run.smithi093.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T18:39:28.593 INFO:teuthology.orchestra.run.smithi093.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T18:39:30.684 INFO:teuthology.orchestra.run.smithi161.stdout:done 2022-03-10T18:39:31.078 DEBUG:teuthology.orchestra.run.smithi161:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T18:39:31.113 INFO:teuthology.orchestra.run.smithi161.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T18:39:31.113 INFO:teuthology.orchestra.run.smithi161.stdout:Status: install ok installed 2022-03-10T18:39:31.113 INFO:teuthology.orchestra.run.smithi161.stdout:Priority: optional 2022-03-10T18:39:31.114 INFO:teuthology.orchestra.run.smithi161.stdout:Section: kernel 2022-03-10T18:39:31.114 INFO:teuthology.orchestra.run.smithi161.stdout:Installed-Size: 19 2022-03-10T18:39:31.114 INFO:teuthology.orchestra.run.smithi161.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T18:39:31.114 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: amd64 2022-03-10T18:39:31.115 INFO:teuthology.orchestra.run.smithi161.stdout:Source: linux-meta-hwe-5.13 2022-03-10T18:39:31.115 INFO:teuthology.orchestra.run.smithi161.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T18:39:31.115 INFO:teuthology.orchestra.run.smithi161.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T18:39:31.115 INFO:teuthology.orchestra.run.smithi161.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T18:39:31.115 INFO:teuthology.orchestra.run.smithi161.stdout:Recommends: thermald 2022-03-10T18:39:31.116 INFO:teuthology.orchestra.run.smithi161.stdout:Description: Generic Linux kernel image 2022-03-10T18:39:31.116 INFO:teuthology.orchestra.run.smithi161.stdout: This package will always depend on the latest generic kernel image 2022-03-10T18:39:31.116 INFO:teuthology.orchestra.run.smithi161.stdout: available. 2022-03-10T18:39:31.116 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T18:39:31.117 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T18:39:31.117 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T18:39:31.117 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-10T18:39:31.118 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi161.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T18:39:31.118 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-get clean 2022-03-10T18:39:31.172 INFO:teuthology.orchestra.run.smithi111.stdout:done 2022-03-10T18:39:31.180 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-get update 2022-03-10T18:39:31.271 INFO:teuthology.orchestra.run.smithi093.stdout:done 2022-03-10T18:39:31.335 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T18:39:31.447 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:39:31.530 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T18:39:31.531 DEBUG:teuthology.orchestra.run.smithi111:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T18:39:31.566 INFO:teuthology.orchestra.run.smithi111.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T18:39:31.567 INFO:teuthology.orchestra.run.smithi111.stdout:Status: install ok installed 2022-03-10T18:39:31.567 INFO:teuthology.orchestra.run.smithi111.stdout:Priority: optional 2022-03-10T18:39:31.567 INFO:teuthology.orchestra.run.smithi111.stdout:Section: kernel 2022-03-10T18:39:31.567 INFO:teuthology.orchestra.run.smithi111.stdout:Installed-Size: 19 2022-03-10T18:39:31.568 INFO:teuthology.orchestra.run.smithi111.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T18:39:31.568 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: amd64 2022-03-10T18:39:31.568 INFO:teuthology.orchestra.run.smithi111.stdout:Source: linux-meta-hwe-5.13 2022-03-10T18:39:31.568 INFO:teuthology.orchestra.run.smithi111.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T18:39:31.568 INFO:teuthology.orchestra.run.smithi111.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T18:39:31.569 INFO:teuthology.orchestra.run.smithi111.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T18:39:31.569 INFO:teuthology.orchestra.run.smithi111.stdout:Recommends: thermald 2022-03-10T18:39:31.569 INFO:teuthology.orchestra.run.smithi111.stdout:Description: Generic Linux kernel image 2022-03-10T18:39:31.569 INFO:teuthology.orchestra.run.smithi111.stdout: This package will always depend on the latest generic kernel image 2022-03-10T18:39:31.570 INFO:teuthology.orchestra.run.smithi111.stdout: available. 2022-03-10T18:39:31.570 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T18:39:31.570 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T18:39:31.571 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T18:39:31.571 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-10T18:39:31.571 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T18:39:31.572 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get clean 2022-03-10T18:39:31.606 DEBUG:teuthology.orchestra.run.smithi093:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T18:39:31.618 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T18:39:31.641 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2022-03-10T18:39:31.671 INFO:teuthology.orchestra.run.smithi093.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T18:39:31.671 INFO:teuthology.orchestra.run.smithi093.stdout:Status: install ok installed 2022-03-10T18:39:31.672 INFO:teuthology.orchestra.run.smithi093.stdout:Priority: optional 2022-03-10T18:39:31.672 INFO:teuthology.orchestra.run.smithi093.stdout:Section: kernel 2022-03-10T18:39:31.672 INFO:teuthology.orchestra.run.smithi093.stdout:Installed-Size: 19 2022-03-10T18:39:31.672 INFO:teuthology.orchestra.run.smithi093.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T18:39:31.672 INFO:teuthology.orchestra.run.smithi093.stdout:Architecture: amd64 2022-03-10T18:39:31.673 INFO:teuthology.orchestra.run.smithi093.stdout:Source: linux-meta-hwe-5.13 2022-03-10T18:39:31.673 INFO:teuthology.orchestra.run.smithi093.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T18:39:31.673 INFO:teuthology.orchestra.run.smithi093.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T18:39:31.673 INFO:teuthology.orchestra.run.smithi093.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T18:39:31.674 INFO:teuthology.orchestra.run.smithi093.stdout:Recommends: thermald 2022-03-10T18:39:31.674 INFO:teuthology.orchestra.run.smithi093.stdout:Description: Generic Linux kernel image 2022-03-10T18:39:31.674 INFO:teuthology.orchestra.run.smithi093.stdout: This package will always depend on the latest generic kernel image 2022-03-10T18:39:31.674 INFO:teuthology.orchestra.run.smithi093.stdout: available. 2022-03-10T18:39:31.675 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T18:39:31.675 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T18:39:31.675 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T18:39:31.675 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-10T18:39:31.676 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi093.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T18:39:31.676 DEBUG:teuthology.orchestra.run.smithi093:> sudo apt-get clean 2022-03-10T18:39:31.750 DEBUG:teuthology.orchestra.run.smithi093:> sudo apt-get update 2022-03-10T18:39:31.808 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T18:39:31.910 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:39:31.993 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T18:39:32.017 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:39:32.017 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T18:39:32.075 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T18:39:32.087 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T18:39:32.170 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T18:39:32.876 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2022-03-10T18:39:32.899 DEBUG:teuthology.orchestra.run.smithi161:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T18:39:32.967 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2022-03-10T18:39:33.140 INFO:teuthology.orchestra.run.smithi161.stdout:Building dependency tree... 2022-03-10T18:39:33.141 INFO:teuthology.orchestra.run.smithi161.stdout:Reading state information... 2022-03-10T18:39:33.323 INFO:teuthology.orchestra.run.smithi161.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T18:39:33.323 INFO:teuthology.orchestra.run.smithi161.stdout:0 upgraded, 0 newly installed, 0 to remove and 96 not upgraded. 2022-03-10T18:39:33.325 DEBUG:teuthology.orchestra.run.smithi161:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T18:39:33.347 INFO:teuthology.orchestra.run.smithi161.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T18:39:33.347 INFO:teuthology.orchestra.run.smithi161.stdout:Status: install ok installed 2022-03-10T18:39:33.347 INFO:teuthology.orchestra.run.smithi161.stdout:Priority: optional 2022-03-10T18:39:33.348 INFO:teuthology.orchestra.run.smithi161.stdout:Section: kernel 2022-03-10T18:39:33.348 INFO:teuthology.orchestra.run.smithi161.stdout:Installed-Size: 19 2022-03-10T18:39:33.348 INFO:teuthology.orchestra.run.smithi161.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T18:39:33.349 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: amd64 2022-03-10T18:39:33.349 INFO:teuthology.orchestra.run.smithi161.stdout:Source: linux-meta-hwe-5.13 2022-03-10T18:39:33.349 INFO:teuthology.orchestra.run.smithi161.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T18:39:33.349 INFO:teuthology.orchestra.run.smithi161.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T18:39:33.350 INFO:teuthology.orchestra.run.smithi161.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T18:39:33.350 INFO:teuthology.orchestra.run.smithi161.stdout:Recommends: thermald 2022-03-10T18:39:33.351 INFO:teuthology.orchestra.run.smithi161.stdout:Description: Generic Linux kernel image 2022-03-10T18:39:33.351 INFO:teuthology.orchestra.run.smithi161.stdout: This package will always depend on the latest generic kernel image 2022-03-10T18:39:33.352 INFO:teuthology.orchestra.run.smithi161.stdout: available. 2022-03-10T18:39:33.352 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2022-03-10T18:39:33.353 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-03-10T18:39:33.358 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.XeV9aV7naS 2022-03-10T18:39:33.359 DEBUG:teuthology.orchestra.run.smithi161:> sudo cp /boot/grub/grub.cfg /tmp/tmp.XeV9aV7naS 2022-03-10T18:39:33.371 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T18:39:33.416 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.XeV9aV7naS 2022-03-10T18:39:33.443 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-03-10T18:39:33.459 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.XeV9aV7naS is 8KB 2022-03-10T18:39:33.474 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.XeV9aV7naS 2022-03-10T18:39:33.482 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T18:39:33.539 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:39:33.540 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T18:39:33.540 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T18:39:33.543 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2022-03-10T18:39:33.566 DEBUG:teuthology.orchestra.run.smithi093:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T18:39:33.577 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T18:39:33.578 DEBUG:teuthology.orchestra.run.smithi161:> sudo update-grub 2022-03-10T18:39:33.621 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-03-10T18:39:33.621 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-03-10T18:39:33.630 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2022-03-10T18:39:33.804 INFO:teuthology.orchestra.run.smithi093.stdout:Building dependency tree... 2022-03-10T18:39:33.805 INFO:teuthology.orchestra.run.smithi111.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T18:39:33.805 INFO:teuthology.orchestra.run.smithi111.stdout:0 upgraded, 0 newly installed, 0 to remove and 96 not upgraded. 2022-03-10T18:39:33.805 INFO:teuthology.orchestra.run.smithi093.stdout:Reading state information... 2022-03-10T18:39:33.806 DEBUG:teuthology.orchestra.run.smithi111:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T18:39:33.828 INFO:teuthology.orchestra.run.smithi111.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T18:39:33.828 INFO:teuthology.orchestra.run.smithi111.stdout:Status: install ok installed 2022-03-10T18:39:33.829 INFO:teuthology.orchestra.run.smithi111.stdout:Priority: optional 2022-03-10T18:39:33.829 INFO:teuthology.orchestra.run.smithi111.stdout:Section: kernel 2022-03-10T18:39:33.829 INFO:teuthology.orchestra.run.smithi111.stdout:Installed-Size: 19 2022-03-10T18:39:33.829 INFO:teuthology.orchestra.run.smithi111.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T18:39:33.829 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: amd64 2022-03-10T18:39:33.830 INFO:teuthology.orchestra.run.smithi111.stdout:Source: linux-meta-hwe-5.13 2022-03-10T18:39:33.830 INFO:teuthology.orchestra.run.smithi111.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T18:39:33.830 INFO:teuthology.orchestra.run.smithi111.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T18:39:33.830 INFO:teuthology.orchestra.run.smithi111.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T18:39:33.831 INFO:teuthology.orchestra.run.smithi111.stdout:Recommends: thermald 2022-03-10T18:39:33.831 INFO:teuthology.orchestra.run.smithi111.stdout:Description: Generic Linux kernel image 2022-03-10T18:39:33.831 INFO:teuthology.orchestra.run.smithi111.stdout: This package will always depend on the latest generic kernel image 2022-03-10T18:39:33.831 INFO:teuthology.orchestra.run.smithi111.stdout: available. 2022-03-10T18:39:33.832 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-03-10T18:39:33.875 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.37ouIrlOYo 2022-03-10T18:39:33.875 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub/grub.cfg /tmp/tmp.37ouIrlOYo 2022-03-10T18:39:33.891 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.37ouIrlOYo 2022-03-10T18:39:33.932 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.37ouIrlOYo is 8KB 2022-03-10T18:39:33.948 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.37ouIrlOYo 2022-03-10T18:39:33.956 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T18:39:33.982 INFO:teuthology.orchestra.run.smithi093.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T18:39:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:0 upgraded, 0 newly installed, 0 to remove and 96 not upgraded. 2022-03-10T18:39:33.986 DEBUG:teuthology.orchestra.run.smithi093:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T18:39:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T18:39:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:Status: install ok installed 2022-03-10T18:39:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:Priority: optional 2022-03-10T18:39:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:Section: kernel 2022-03-10T18:39:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:Installed-Size: 19 2022-03-10T18:39:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T18:39:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:Architecture: amd64 2022-03-10T18:39:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:Source: linux-meta-hwe-5.13 2022-03-10T18:39:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T18:39:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T18:39:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T18:39:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:Recommends: thermald 2022-03-10T18:39:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:Description: Generic Linux kernel image 2022-03-10T18:39:34.009 INFO:teuthology.orchestra.run.smithi093.stdout: This package will always depend on the latest generic kernel image 2022-03-10T18:39:34.009 INFO:teuthology.orchestra.run.smithi093.stdout: available. 2022-03-10T18:39:34.010 DEBUG:teuthology.orchestra.run.smithi093:> mktemp 2022-03-10T18:39:34.012 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:39:34.012 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T18:39:34.012 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T18:39:34.047 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T18:39:34.047 DEBUG:teuthology.orchestra.run.smithi111:> sudo update-grub 2022-03-10T18:39:34.060 INFO:teuthology.orchestra.run.smithi093.stdout:/tmp/tmp.QTcMF1dsmt 2022-03-10T18:39:34.060 DEBUG:teuthology.orchestra.run.smithi093:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QTcMF1dsmt 2022-03-10T18:39:34.076 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 0666 /tmp/tmp.QTcMF1dsmt 2022-03-10T18:39:34.166 DEBUG:teuthology.orchestra.remote:smithi093:/tmp/tmp.QTcMF1dsmt is 8KB 2022-03-10T18:39:34.181 DEBUG:teuthology.orchestra.run.smithi093:> rm -fr /tmp/tmp.QTcMF1dsmt 2022-03-10T18:39:34.188 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T18:39:34.243 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:39:34.243 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T18:39:34.243 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T18:39:34.287 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T18:39:34.288 DEBUG:teuthology.orchestra.run.smithi093:> sudo update-grub 2022-03-10T18:39:34.764 INFO:teuthology.orchestra.run.smithi161.stderr:Sourcing file `/etc/default/grub' 2022-03-10T18:39:34.765 INFO:teuthology.orchestra.run.smithi161.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T18:39:34.783 INFO:teuthology.orchestra.run.smithi161.stderr:Generating grub configuration file ... 2022-03-10T18:39:35.017 INFO:teuthology.orchestra.run.smithi161.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T18:39:35.029 INFO:teuthology.orchestra.run.smithi161.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T18:39:35.219 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub' 2022-03-10T18:39:35.220 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T18:39:35.229 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2022-03-10T18:39:35.364 INFO:teuthology.orchestra.run.smithi161.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T18:39:35.376 INFO:teuthology.orchestra.run.smithi161.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T18:39:35.476 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T18:39:35.483 INFO:teuthology.orchestra.run.smithi093.stderr:Sourcing file `/etc/default/grub' 2022-03-10T18:39:35.487 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T18:39:35.494 INFO:teuthology.orchestra.run.smithi093.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T18:39:35.511 INFO:teuthology.orchestra.run.smithi093.stderr:Generating grub configuration file ... 2022-03-10T18:39:35.742 INFO:teuthology.orchestra.run.smithi093.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T18:39:35.754 INFO:teuthology.orchestra.run.smithi093.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T18:39:35.867 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T18:39:35.879 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T18:39:36.138 INFO:teuthology.orchestra.run.smithi093.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T18:39:36.150 INFO:teuthology.orchestra.run.smithi093.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T18:39:36.955 INFO:teuthology.orchestra.run.smithi161.stderr:done 2022-03-10T18:39:36.957 DEBUG:teuthology.orchestra.run.smithi161:> sudo shutdown -r now 2022-03-10T18:39:37.487 INFO:teuthology.orchestra.run.smithi111.stderr:done 2022-03-10T18:39:37.488 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2022-03-10T18:39:37.785 INFO:teuthology.orchestra.run.smithi093.stderr:done 2022-03-10T18:39:37.786 DEBUG:teuthology.orchestra.run.smithi093:> sudo shutdown -r now 2022-03-10T18:40:06.984 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-10T18:40:06.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-03-10T18:40:06.986 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:40:07.492 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-10T18:40:07.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-03-10T18:40:07.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:40:07.790 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-10T18:40:07.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2022-03-10T18:40:07.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:40:26.315 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.93 2022-03-10T18:40:41.675 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2022-03-10T18:40:42.187 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-03-10T18:40:56.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2022-03-10T18:40:56.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:41:11.627 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.93 2022-03-10T18:41:11.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-03-10T18:41:11.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:41:12.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-03-10T18:41:12.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:41:27.291 DEBUG:teuthology.orchestra.run.smithi161:> true 2022-03-10T18:41:27.499 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-03-10T18:41:28.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-03-10T18:41:28.117 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-35-generic"... 2022-03-10T18:41:28.118 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2022-03-10T18:41:28.124 INFO:teuthology.orchestra.run.smithi161.stdout:5.13.0-35-generic 2022-03-10T18:41:28.124 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T18:41:28.124 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T18:41:28.125 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T18:41:29.126 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-10T18:41:29.127 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T18:41:29.177 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2022-03-10T18:41:29.201 DEBUG:teuthology.parallel:result is None 2022-03-10T18:41:41.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2022-03-10T18:41:41.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:41:41.806 DEBUG:teuthology.orchestra.run.smithi093:> true 2022-03-10T18:41:42.363 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi093.front.sepia.ceph.com' 2022-03-10T18:41:42.363 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-10T18:41:42.364 DEBUG:teuthology.orchestra.run.smithi093:> uname -r 2022-03-10T18:41:42.411 INFO:teuthology.orchestra.run.smithi093.stdout:5.13.0-35-generic 2022-03-10T18:41:42.411 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T18:41:42.412 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T18:41:42.412 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T18:41:43.413 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-10T18:41:43.414 DEBUG:teuthology.orchestra.run.smithi093:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T18:41:43.476 INFO:teuthology.orchestra.run.smithi093.stdout:ttyS1 2022-03-10T18:41:43.499 DEBUG:teuthology.parallel:result is None 2022-03-10T18:41:57.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-03-10T18:41:57.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:41:57.613 DEBUG:teuthology.orchestra.run.smithi111:> true 2022-03-10T18:41:58.099 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-03-10T18:41:58.099 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-35-generic"... 2022-03-10T18:41:58.100 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2022-03-10T18:41:58.146 INFO:teuthology.orchestra.run.smithi111.stdout:5.13.0-35-generic 2022-03-10T18:41:58.146 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T18:41:58.147 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T18:41:58.147 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T18:41:59.148 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-10T18:41:59.148 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T18:41:59.165 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2022-03-10T18:41:59.194 DEBUG:teuthology.parallel:result is None 2022-03-10T18:41:59.195 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-10T18:41:59.203 INFO:teuthology.task.internal:Creating test directory... 2022-03-10T18:41:59.204 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T18:41:59.207 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T18:41:59.238 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T18:41:59.244 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-10T18:41:59.253 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-10T18:41:59.304 INFO:teuthology.task.internal:Creating archive directory... 2022-03-10T18:41:59.304 DEBUG:teuthology.orchestra.run.smithi093:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T18:41:59.307 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T18:41:59.311 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T18:41:59.444 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-10T18:41:59.452 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-10T18:41:59.453 DEBUG:teuthology.orchestra.run.smithi093:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T18:41:59.456 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T18:41:59.458 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T18:41:59.479 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T18:41:59.483 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T18:41:59.484 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T18:41:59.488 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T18:41:59.490 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T18:41:59.492 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T18:41:59.494 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-10T18:41:59.503 INFO:teuthology.task.internal:Configuring sudo... 2022-03-10T18:41:59.503 DEBUG:teuthology.orchestra.run.smithi093:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T18:41:59.539 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T18:41:59.542 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T18:41:59.557 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-10T18:41:59.568 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-10T18:41:59.569 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T18:41:59.572 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T18:41:59.575 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T18:41:59.584 DEBUG:teuthology.orchestra.run.smithi093:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T18:41:59.628 DEBUG:teuthology.orchestra.run.smithi093:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T18:41:59.634 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:41:59.634 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T18:41:59.652 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T18:41:59.663 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T18:41:59.669 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:41:59.669 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T18:41:59.691 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T18:41:59.701 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T18:41:59.707 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:41:59.707 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T18:41:59.777 DEBUG:teuthology.orchestra.run.smithi093:> sudo service rsyslog restart 2022-03-10T18:41:59.780 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2022-03-10T18:41:59.783 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2022-03-10T18:41:59.835 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-10T18:41:59.844 INFO:teuthology.task.internal:Starting timer... 2022-03-10T18:41:59.845 INFO:teuthology.run_tasks:Running task pcp... 2022-03-10T18:41:59.898 INFO:teuthology.run_tasks:Running task selinux... 2022-03-10T18:41:59.955 DEBUG:teuthology.task.selinux:Excluding smithi093: OS 'ubuntu' does not support SELinux 2022-03-10T18:41:59.956 DEBUG:teuthology.task.selinux:Excluding smithi111: OS 'ubuntu' does not support SELinux 2022-03-10T18:41:59.956 DEBUG:teuthology.task.selinux:Excluding smithi161: OS 'ubuntu' does not support SELinux 2022-03-10T18:41:59.956 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-10T18:41:59.957 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-10T18:41:59.957 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-10T18:41:59.957 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-10T18:42:00.006 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-10T18:42:00.007 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-10T18:42:00.032 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-10T18:42:00.034 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi093.front.sepia.ceph.com,smithi111.front.sepia.ceph.com,smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-10T18:45:04.414 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi093.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi161.front.sepia.ceph.com')] 2022-03-10T18:45:04.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2022-03-10T18:45:04.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:45:04.463 DEBUG:teuthology.orchestra.run.smithi093:> true 2022-03-10T18:45:04.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-03-10T18:45:04.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:45:04.623 DEBUG:teuthology.orchestra.run.smithi111:> true 2022-03-10T18:45:04.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-03-10T18:45:04.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T18:45:04.798 DEBUG:teuthology.orchestra.run.smithi161:> true 2022-03-10T18:45:04.920 INFO:teuthology.run_tasks:Running task clock... 2022-03-10T18:45:04.933 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-10T18:45:04.934 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T18:45:04.936 DEBUG:teuthology.orchestra.run.smithi093:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T18:45:04.939 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T18:45:04.939 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T18:45:04.942 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T18:45:04.942 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T18:45:04.969 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T18:45:04.970 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: Command line: ntpd -gq 2022-03-10T18:45:04.970 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: proto: precision = 0.092 usec (-23) 2022-03-10T18:45:04.971 INFO:teuthology.orchestra.run.smithi093.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T18:45:04.972 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T18:45:04.972 INFO:teuthology.orchestra.run.smithi093.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T18:45:04.973 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T18:45:04.974 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T18:45:04.974 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T18:45:04.974 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T18:45:04.974 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: Listen normally on 3 enp3s0f1 172.21.15.93:123 2022-03-10T18:45:04.975 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: Listen normally on 4 lo [::1]:123 2022-03-10T18:45:04.975 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d635%5]:123 2022-03-10T18:45:04.975 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:04 ntpd[9062]: Listening on routing socket on fd #22 for interface updates 2022-03-10T18:45:04.976 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T18:45:04.976 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: Command line: ntpd -gq 2022-03-10T18:45:04.977 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: proto: precision = 0.072 usec (-24) 2022-03-10T18:45:04.977 INFO:teuthology.orchestra.run.smithi111.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T18:45:04.977 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T18:45:04.978 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T18:45:04.978 INFO:teuthology.orchestra.run.smithi111.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T18:45:04.979 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T18:45:04.980 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T18:45:04.980 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T18:45:04.980 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: Listen normally on 3 enp3s0f1 172.21.15.111:123 2022-03-10T18:45:04.980 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: Listen normally on 4 lo [::1]:123 2022-03-10T18:45:04.980 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc2f%5]:123 2022-03-10T18:45:04.981 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:04 ntpd[9093]: Listening on routing socket on fd #22 for interface updates 2022-03-10T18:45:04.997 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T18:45:04.997 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: Command line: ntpd -gq 2022-03-10T18:45:04.998 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: proto: precision = 0.100 usec (-23) 2022-03-10T18:45:04.998 INFO:teuthology.orchestra.run.smithi161.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T18:45:04.999 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T18:45:04.999 INFO:teuthology.orchestra.run.smithi161.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T18:45:05.001 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T18:45:05.001 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T18:45:05.001 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T18:45:05.002 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T18:45:05.002 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: Listen normally on 3 enp3s0f1 172.21.15.161:123 2022-03-10T18:45:05.002 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: Listen normally on 4 lo [::1]:123 2022-03-10T18:45:05.002 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7251%5]:123 2022-03-10T18:45:05.003 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:04 ntpd[9026]: Listening on routing socket on fd #22 for interface updates 2022-03-10T18:45:05.971 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:05.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:05.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:05.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:05.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:05.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:05.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:05.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:05.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:05.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:05.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:05.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:05 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:05 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:05 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:05 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:05 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:05 ntpd[9093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:05 ntpd[9093]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:05.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:05 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:05.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:05 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:05.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:05 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:05.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:05 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:05.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:05 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:05.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:05 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:06.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:06 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:06.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:06 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:06.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:06 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:06.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:06 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:06.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:06 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:06.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:06 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:06.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:06 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:06.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:06 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:06.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:06 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:06 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:06 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:07.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:06 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:07.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:07.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:07.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:07.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:07.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.976 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.976 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:07.976 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:07 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:07.977 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:07 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.977 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:07 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.977 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:07 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.978 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:07 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.978 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:07 ntpd[9093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:07.978 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:07 ntpd[9093]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:07.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:07 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:07 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:07 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:07.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:07 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:07.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:07 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:07.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:07 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:08.971 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:08 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:08.971 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:08 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:08.971 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:08 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:08.971 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:08 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:08.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:08 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:08.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:08 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:08.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:08.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:08 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.976 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.976 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:09 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.978 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.979 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.979 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.979 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.979 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.979 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.980 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.980 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.980 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.980 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.980 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.981 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.981 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.981 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.981 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.982 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.982 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.982 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.982 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.982 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.983 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.983 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.983 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.983 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:09 ntpd[9093]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:09.997 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:09 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:09 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:09 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:09.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:09 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:09.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:09 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:09.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:09 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:10.970 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:10 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:10.971 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:10 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:10.971 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:10 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:10.971 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:10 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:10.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:10 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:10.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:10 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:10.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:10.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.000 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:11.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:11.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:11.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:11.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.002 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.003 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:11.003 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:10 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:11.972 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.973 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.974 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.975 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.976 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:11.976 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:11.977 INFO:teuthology.orchestra.run.smithi093.stdout:10 Mar 18:45:11 ntpd[9062]: ntpd: time slew -0.012245 s 2022-03-10T18:45:11.978 INFO:teuthology.orchestra.run.smithi093.stdout:ntpd: time slew -0.012245s 2022-03-10T18:45:11.978 INFO:teuthology.orchestra.run.smithi093.stderr:10 Mar 18:45:11 ntpd[9062]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T18:45:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.982 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.982 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.984 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:11.984 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:11.984 INFO:teuthology.orchestra.run.smithi111.stdout:10 Mar 18:45:11 ntpd[9093]: ntpd: time slew -0.000485 s 2022-03-10T18:45:11.985 INFO:teuthology.orchestra.run.smithi111.stdout:ntpd: time slew -0.000485s 2022-03-10T18:45:11.985 INFO:teuthology.orchestra.run.smithi111.stderr:10 Mar 18:45:11 ntpd[9093]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T18:45:11.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:11 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.998 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:11 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:11 ntpd[9026]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T18:45:11.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:11 ntpd[9026]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T18:45:11.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:11 ntpd[9026]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T18:45:11.999 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:11 ntpd[9026]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T18:45:12.000 INFO:teuthology.orchestra.run.smithi161.stdout:10 Mar 18:45:11 ntpd[9026]: ntpd: time slew +0.001119 s 2022-03-10T18:45:12.000 INFO:teuthology.orchestra.run.smithi161.stdout:ntpd: time slew +0.001119s 2022-03-10T18:45:12.001 INFO:teuthology.orchestra.run.smithi161.stderr:10 Mar 18:45:11 ntpd[9026]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T18:45:12.033 INFO:teuthology.orchestra.run.smithi093.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T18:45:12.034 INFO:teuthology.orchestra.run.smithi093.stdout:============================================================================== 2022-03-10T18:45:12.034 INFO:teuthology.orchestra.run.smithi093.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.035 INFO:teuthology.orchestra.run.smithi093.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.035 INFO:teuthology.orchestra.run.smithi093.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.036 INFO:teuthology.orchestra.run.smithi093.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.071 INFO:teuthology.orchestra.run.smithi161.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T18:45:12.072 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================== 2022-03-10T18:45:12.072 INFO:teuthology.orchestra.run.smithi161.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.072 INFO:teuthology.orchestra.run.smithi161.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.072 INFO:teuthology.orchestra.run.smithi161.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.072 INFO:teuthology.orchestra.run.smithi161.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.112 INFO:teuthology.orchestra.run.smithi111.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T18:45:12.112 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================== 2022-03-10T18:45:12.112 INFO:teuthology.orchestra.run.smithi111.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.113 INFO:teuthology.orchestra.run.smithi111.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.113 INFO:teuthology.orchestra.run.smithi111.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.113 INFO:teuthology.orchestra.run.smithi111.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T18:45:12.114 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-10T18:45:12.141 INFO:tasks.kubeadm:Kubeadm start 2022-03-10T18:45:12.141 INFO:tasks.kubeadm:Config: {'version': '1.21'} 2022-03-10T18:45:12.142 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T18:45:12.145 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T18:45:12.158 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T18:45:12.165 INFO:teuthology.orchestra.run.smithi093.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T18:45:12.175 INFO:teuthology.orchestra.run.smithi093.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T18:45:12.182 INFO:teuthology.orchestra.run.smithi111.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T18:45:12.183 INFO:teuthology.orchestra.run.smithi093.stdout:net.ipv4.ip_forward = 1 2022-03-10T18:45:12.185 INFO:teuthology.orchestra.run.smithi161.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T18:45:12.193 INFO:teuthology.orchestra.run.smithi111.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T18:45:12.194 INFO:teuthology.orchestra.run.smithi161.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T18:45:12.202 INFO:teuthology.orchestra.run.smithi111.stdout:net.ipv4.ip_forward = 1 2022-03-10T18:45:12.203 INFO:teuthology.orchestra.run.smithi161.stdout:net.ipv4.ip_forward = 1 2022-03-10T18:45:12.263 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:45:12.264 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T18:45:12.279 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:45:12.279 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/docker/daemon.json 2022-03-10T18:45:12.292 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:45:12.293 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T18:45:12.307 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:45:12.308 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/docker/daemon.json 2022-03-10T18:45:12.319 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:45:12.319 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T18:45:12.334 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:45:12.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/docker/daemon.json 2022-03-10T18:45:12.394 DEBUG:teuthology.orchestra.run.smithi093:> sudo systemctl restart docker || true 2022-03-10T18:45:12.396 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl restart docker || true 2022-03-10T18:45:12.399 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl restart docker || true 2022-03-10T18:45:13.385 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-10T18:45:13.392 INFO:tasks.kubeadm:172.21.15.93 is in 172.21.0.0/20 at pos 3932 2022-03-10T18:45:13.406 INFO:tasks.kubeadm:Pod subnet: 10.250.224.0/21 2022-03-10T18:45:13.406 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-10T18:45:13.406 DEBUG:teuthology.orchestra.run.smithi093:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T18:45:13.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T18:45:13.412 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T18:45:13.435 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-03-10T18:45:13.435 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:13.436 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-03-10T18:45:13.437 INFO:teuthology.orchestra.run.smithi111.stderr: 2022-03-10T18:45:13.437 INFO:teuthology.orchestra.run.smithi111.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:13.438 INFO:teuthology.orchestra.run.smithi111.stderr: 2022-03-10T18:45:13.448 INFO:teuthology.orchestra.run.smithi093.stderr: 2022-03-10T18:45:13.448 INFO:teuthology.orchestra.run.smithi093.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:13.448 INFO:teuthology.orchestra.run.smithi093.stderr: 2022-03-10T18:45:13.553 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T18:45:13.566 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T18:45:13.666 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:45:13.705 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:45:13.705 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T18:45:13.728 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:45:13.748 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T18:45:13.775 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T18:45:13.810 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T18:45:13.830 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T18:45:13.857 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T18:45:13.892 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T18:45:15.044 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-03-10T18:45:15.079 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2022-03-10T18:45:15.177 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2022-03-10T18:45:15.213 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-03-10T18:45:15.214 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-03-10T18:45:15.249 INFO:teuthology.orchestra.run.smithi093.stdout:Building dependency tree... 2022-03-10T18:45:15.250 INFO:teuthology.orchestra.run.smithi093.stdout:Reading state information... 2022-03-10T18:45:15.306 INFO:teuthology.orchestra.run.smithi111.stdout:89 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T18:45:15.324 INFO:teuthology.orchestra.run.smithi111.stderr: 2022-03-10T18:45:15.325 INFO:teuthology.orchestra.run.smithi111.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:15.325 INFO:teuthology.orchestra.run.smithi111.stderr: 2022-03-10T18:45:15.337 INFO:teuthology.orchestra.run.smithi093.stdout:89 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T18:45:15.350 INFO:teuthology.orchestra.run.smithi161.stdout:Building dependency tree... 2022-03-10T18:45:15.351 INFO:teuthology.orchestra.run.smithi161.stdout:Reading state information... 2022-03-10T18:45:15.351 INFO:teuthology.orchestra.run.smithi093.stderr: 2022-03-10T18:45:15.352 INFO:teuthology.orchestra.run.smithi093.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:15.352 INFO:teuthology.orchestra.run.smithi093.stderr: 2022-03-10T18:45:15.379 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-03-10T18:45:15.404 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2022-03-10T18:45:15.439 INFO:teuthology.orchestra.run.smithi161.stdout:89 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T18:45:15.454 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-03-10T18:45:15.454 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:15.454 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-03-10T18:45:15.507 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2022-03-10T18:45:15.551 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-03-10T18:45:15.552 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-03-10T18:45:15.577 INFO:teuthology.orchestra.run.smithi093.stdout:Building dependency tree... 2022-03-10T18:45:15.579 INFO:teuthology.orchestra.run.smithi093.stdout:Reading state information... 2022-03-10T18:45:15.666 INFO:teuthology.orchestra.run.smithi161.stdout:Building dependency tree... 2022-03-10T18:45:15.667 INFO:teuthology.orchestra.run.smithi161.stdout:Reading state information... 2022-03-10T18:45:15.731 INFO:teuthology.orchestra.run.smithi111.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T18:45:15.731 INFO:teuthology.orchestra.run.smithi111.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T18:45:15.732 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2022-03-10T18:45:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: apt-transport-https 2022-03-10T18:45:15.743 INFO:teuthology.orchestra.run.smithi093.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T18:45:15.743 INFO:teuthology.orchestra.run.smithi093.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T18:45:15.743 INFO:teuthology.orchestra.run.smithi093.stdout:The following NEW packages will be installed: 2022-03-10T18:45:15.744 INFO:teuthology.orchestra.run.smithi093.stdout: apt-transport-https 2022-03-10T18:45:15.828 INFO:teuthology.orchestra.run.smithi161.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T18:45:15.829 INFO:teuthology.orchestra.run.smithi161.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T18:45:15.829 INFO:teuthology.orchestra.run.smithi161.stdout:The following NEW packages will be installed: 2022-03-10T18:45:15.830 INFO:teuthology.orchestra.run.smithi161.stdout: apt-transport-https 2022-03-10T18:45:15.914 INFO:teuthology.orchestra.run.smithi111.stdout:0 upgraded, 1 newly installed, 0 to remove and 89 not upgraded. 2022-03-10T18:45:15.915 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 4,680 B of archives. 2022-03-10T18:45:15.915 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T18:45:15.915 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T18:45:15.928 INFO:teuthology.orchestra.run.smithi093.stdout:0 upgraded, 1 newly installed, 0 to remove and 89 not upgraded. 2022-03-10T18:45:15.929 INFO:teuthology.orchestra.run.smithi093.stdout:Need to get 4,680 B of archives. 2022-03-10T18:45:15.929 INFO:teuthology.orchestra.run.smithi093.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T18:45:15.929 INFO:teuthology.orchestra.run.smithi093.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T18:45:16.015 INFO:teuthology.orchestra.run.smithi161.stdout:0 upgraded, 1 newly installed, 0 to remove and 89 not upgraded. 2022-03-10T18:45:16.016 INFO:teuthology.orchestra.run.smithi161.stdout:Need to get 4,680 B of archives. 2022-03-10T18:45:16.016 INFO:teuthology.orchestra.run.smithi161.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T18:45:16.016 INFO:teuthology.orchestra.run.smithi161.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T18:45:16.141 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T18:45:16.142 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T18:45:16.142 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: falling back to frontend: Readline 2022-03-10T18:45:16.145 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T18:45:16.145 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T18:45:16.145 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: falling back to frontend: Readline 2022-03-10T18:45:16.163 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T18:45:16.163 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T18:45:16.163 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: falling back to frontend: Teletype 2022-03-10T18:45:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T18:45:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T18:45:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: falling back to frontend: Teletype 2022-03-10T18:45:16.169 INFO:teuthology.orchestra.run.smithi093.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T18:45:16.174 INFO:teuthology.orchestra.run.smithi111.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T18:45:16.216 INFO:teuthology.orchestra.run.smithi093.stdout:Fetched 4,680 B in 0s (26.5 kB/s) 2022-03-10T18:45:16.226 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 4,680 B in 0s (26.8 kB/s) 2022-03-10T18:45:16.239 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T18:45:16.239 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T18:45:16.240 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: falling back to frontend: Readline 2022-03-10T18:45:16.259 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T18:45:16.259 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T18:45:16.259 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: falling back to frontend: Teletype 2022-03-10T18:45:16.265 INFO:teuthology.orchestra.run.smithi161.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T18:45:16.310 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T18:45:16.316 INFO:teuthology.orchestra.run.smithi161.stdout:Fetched 4,680 B in 0s (26.4 kB/s) 2022-03-10T18:45:16.321 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T18:45:16.351 INFO:teuthology.orchestra.run.smithi093.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T18:45:16.353 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T18:45:16.365 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T18:45:16.369 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T18:45:16.383 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T18:45:16.406 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T18:45:16.411 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T18:45:16.451 INFO:teuthology.orchestra.run.smithi161.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T18:45:16.455 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T18:45:16.496 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T18:45:16.653 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T18:45:16.688 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T18:45:16.755 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T18:45:17.210 INFO:teuthology.orchestra.run.smithi093.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T18:45:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T18:45:17.224 INFO:teuthology.orchestra.run.smithi093.stderr: 2022-03-10T18:45:17.225 INFO:teuthology.orchestra.run.smithi093.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:17.225 INFO:teuthology.orchestra.run.smithi093.stderr: 2022-03-10T18:45:17.235 INFO:teuthology.orchestra.run.smithi111.stderr: 2022-03-10T18:45:17.236 INFO:teuthology.orchestra.run.smithi111.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:17.236 INFO:teuthology.orchestra.run.smithi111.stderr: 2022-03-10T18:45:17.258 INFO:teuthology.orchestra.run.smithi161.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T18:45:17.274 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-03-10T18:45:17.274 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:17.274 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-03-10T18:45:17.352 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T18:45:17.464 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:45:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T18:45:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:45:17.527 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T18:45:17.528 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T18:45:17.546 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T18:45:17.561 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T18:45:17.597 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T18:45:17.628 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T18:45:17.643 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T18:45:17.683 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T18:45:18.283 INFO:teuthology.orchestra.run.smithi093.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T18:45:18.384 INFO:teuthology.orchestra.run.smithi093.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T18:45:18.417 INFO:teuthology.orchestra.run.smithi161.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T18:45:18.485 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T18:45:19.237 INFO:teuthology.orchestra.run.smithi093.stdout:Fetched 63.3 kB in 1s (55.1 kB/s) 2022-03-10T18:45:19.237 INFO:teuthology.orchestra.run.smithi161.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T18:45:19.237 INFO:teuthology.orchestra.run.smithi161.stdout:Fetched 63.3 kB in 1s (49.4 kB/s) 2022-03-10T18:45:19.238 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T18:45:19.238 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 63.3 kB in 1s (47.6 kB/s) 2022-03-10T18:45:19.358 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2022-03-10T18:45:19.533 INFO:teuthology.orchestra.run.smithi093.stdout:Building dependency tree... 2022-03-10T18:45:19.534 INFO:teuthology.orchestra.run.smithi093.stdout:Reading state information... 2022-03-10T18:45:19.563 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-03-10T18:45:19.566 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2022-03-10T18:45:19.626 INFO:teuthology.orchestra.run.smithi093.stdout:89 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T18:45:19.639 INFO:teuthology.orchestra.run.smithi093.stderr: 2022-03-10T18:45:19.640 INFO:teuthology.orchestra.run.smithi093.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:19.640 INFO:teuthology.orchestra.run.smithi093.stderr: 2022-03-10T18:45:19.695 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2022-03-10T18:45:19.734 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-03-10T18:45:19.734 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-03-10T18:45:19.737 INFO:teuthology.orchestra.run.smithi161.stdout:Building dependency tree... 2022-03-10T18:45:19.737 INFO:teuthology.orchestra.run.smithi161.stdout:Reading state information... 2022-03-10T18:45:19.821 INFO:teuthology.orchestra.run.smithi111.stdout:89 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T18:45:19.828 INFO:teuthology.orchestra.run.smithi161.stdout:89 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T18:45:19.830 INFO:teuthology.orchestra.run.smithi111.stderr: 2022-03-10T18:45:19.830 INFO:teuthology.orchestra.run.smithi111.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:19.831 INFO:teuthology.orchestra.run.smithi111.stderr: 2022-03-10T18:45:19.844 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-03-10T18:45:19.844 INFO:teuthology.orchestra.run.smithi161.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T18:45:19.845 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-03-10T18:45:19.869 INFO:teuthology.orchestra.run.smithi093.stdout:Building dependency tree... 2022-03-10T18:45:19.870 INFO:teuthology.orchestra.run.smithi093.stdout:Reading state information... 2022-03-10T18:45:19.886 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-03-10T18:45:19.900 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2022-03-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi093.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi093.stdout:bridge-utils set to manually installed. 2022-03-10T18:45:20.021 INFO:teuthology.orchestra.run.smithi093.stdout:The following additional packages will be installed: 2022-03-10T18:45:20.022 INFO:teuthology.orchestra.run.smithi093.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T18:45:20.022 INFO:teuthology.orchestra.run.smithi093.stdout:Suggested packages: 2022-03-10T18:45:20.023 INFO:teuthology.orchestra.run.smithi093.stdout: nftables 2022-03-10T18:45:20.065 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-03-10T18:45:20.066 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-03-10T18:45:20.072 INFO:teuthology.orchestra.run.smithi093.stdout:The following NEW packages will be installed: 2022-03-10T18:45:20.073 INFO:teuthology.orchestra.run.smithi093.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T18:45:20.077 INFO:teuthology.orchestra.run.smithi161.stdout:Building dependency tree... 2022-03-10T18:45:20.078 INFO:teuthology.orchestra.run.smithi161.stdout:Reading state information... 2022-03-10T18:45:20.215 INFO:teuthology.orchestra.run.smithi111.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T18:45:20.215 INFO:teuthology.orchestra.run.smithi111.stdout:bridge-utils set to manually installed. 2022-03-10T18:45:20.215 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2022-03-10T18:45:20.216 INFO:teuthology.orchestra.run.smithi111.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T18:45:20.217 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2022-03-10T18:45:20.217 INFO:teuthology.orchestra.run.smithi111.stdout: nftables 2022-03-10T18:45:20.218 INFO:teuthology.orchestra.run.smithi161.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T18:45:20.218 INFO:teuthology.orchestra.run.smithi161.stdout:bridge-utils set to manually installed. 2022-03-10T18:45:20.219 INFO:teuthology.orchestra.run.smithi161.stdout:The following additional packages will be installed: 2022-03-10T18:45:20.219 INFO:teuthology.orchestra.run.smithi161.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T18:45:20.220 INFO:teuthology.orchestra.run.smithi161.stdout:Suggested packages: 2022-03-10T18:45:20.220 INFO:teuthology.orchestra.run.smithi161.stdout: nftables 2022-03-10T18:45:20.263 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2022-03-10T18:45:20.263 INFO:teuthology.orchestra.run.smithi111.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T18:45:20.265 INFO:teuthology.orchestra.run.smithi161.stdout:The following NEW packages will be installed: 2022-03-10T18:45:20.266 INFO:teuthology.orchestra.run.smithi161.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T18:45:20.267 INFO:teuthology.orchestra.run.smithi093.stdout:0 upgraded, 8 newly installed, 0 to remove and 89 not upgraded. 2022-03-10T18:45:20.267 INFO:teuthology.orchestra.run.smithi093.stdout:Need to get 73.6 MB of archives. 2022-03-10T18:45:20.267 INFO:teuthology.orchestra.run.smithi093.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T18:45:20.267 INFO:teuthology.orchestra.run.smithi093.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T18:45:20.317 INFO:teuthology.orchestra.run.smithi093.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T18:45:20.456 INFO:teuthology.orchestra.run.smithi111.stdout:0 upgraded, 8 newly installed, 0 to remove and 89 not upgraded. 2022-03-10T18:45:20.457 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 73.6 MB of archives. 2022-03-10T18:45:20.457 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T18:45:20.457 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T18:45:20.459 INFO:teuthology.orchestra.run.smithi093.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T18:45:20.459 INFO:teuthology.orchestra.run.smithi161.stdout:0 upgraded, 8 newly installed, 0 to remove and 89 not upgraded. 2022-03-10T18:45:20.460 INFO:teuthology.orchestra.run.smithi161.stdout:Need to get 73.6 MB of archives. 2022-03-10T18:45:20.460 INFO:teuthology.orchestra.run.smithi161.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T18:45:20.461 INFO:teuthology.orchestra.run.smithi161.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T18:45:20.488 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T18:45:20.535 INFO:teuthology.orchestra.run.smithi161.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T18:45:20.603 INFO:teuthology.orchestra.run.smithi093.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T18:45:20.639 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T18:45:20.640 INFO:teuthology.orchestra.run.smithi161.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T18:45:20.741 INFO:teuthology.orchestra.run.smithi161.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T18:45:20.748 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T18:45:20.749 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T18:45:20.803 INFO:teuthology.orchestra.run.smithi093.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T18:45:20.922 INFO:teuthology.orchestra.run.smithi161.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T18:45:21.487 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T18:45:21.582 INFO:teuthology.orchestra.run.smithi161.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T18:45:21.629 INFO:teuthology.orchestra.run.smithi093.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T18:45:21.995 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T18:45:22.033 INFO:teuthology.orchestra.run.smithi161.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T18:45:22.152 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T18:45:22.205 INFO:teuthology.orchestra.run.smithi161.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T18:45:22.352 INFO:teuthology.orchestra.run.smithi093.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T18:45:22.511 INFO:teuthology.orchestra.run.smithi093.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T18:45:22.640 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T18:45:22.640 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T18:45:22.641 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: falling back to frontend: Readline 2022-03-10T18:45:22.648 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T18:45:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T18:45:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:debconf: falling back to frontend: Teletype 2022-03-10T18:45:22.649 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T18:45:22.650 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T18:45:22.650 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: falling back to frontend: Readline 2022-03-10T18:45:22.654 INFO:teuthology.orchestra.run.smithi111.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T18:45:22.657 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T18:45:22.657 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T18:45:22.657 INFO:teuthology.orchestra.run.smithi161.stderr:debconf: falling back to frontend: Teletype 2022-03-10T18:45:22.664 INFO:teuthology.orchestra.run.smithi161.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T18:45:22.703 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 73.6 MB in 2s (34.2 MB/s) 2022-03-10T18:45:22.708 INFO:teuthology.orchestra.run.smithi161.stdout:Fetched 73.6 MB in 2s (34.2 MB/s) 2022-03-10T18:45:22.787 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package conntrack. 2022-03-10T18:45:22.830 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T18:45:22.832 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package conntrack. 2022-03-10T18:45:22.833 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T18:45:22.877 INFO:teuthology.orchestra.run.smithi161.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T18:45:22.878 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T18:45:22.880 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T18:45:22.923 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T18:45:22.924 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T18:45:22.925 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: falling back to frontend: Readline 2022-03-10T18:45:22.931 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T18:45:22.932 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T18:45:22.932 INFO:teuthology.orchestra.run.smithi093.stderr:debconf: falling back to frontend: Teletype 2022-03-10T18:45:22.938 INFO:teuthology.orchestra.run.smithi093.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T18:45:22.947 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T18:45:22.988 INFO:teuthology.orchestra.run.smithi093.stdout:Fetched 73.6 MB in 3s (28.0 MB/s) 2022-03-10T18:45:23.076 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package conntrack. 2022-03-10T18:45:23.119 INFO:teuthology.orchestra.run.smithi093.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T18:45:23.122 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T18:45:23.166 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T18:45:23.181 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package cri-tools. 2022-03-10T18:45:23.193 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T18:45:23.239 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T18:45:23.261 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package cri-tools. 2022-03-10T18:45:23.277 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T18:45:23.311 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T18:45:23.481 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package cri-tools. 2022-03-10T18:45:23.495 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T18:45:23.540 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T18:45:24.262 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ebtables. 2022-03-10T18:45:24.275 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T18:45:24.323 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T18:45:24.364 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ebtables. 2022-03-10T18:45:24.377 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T18:45:24.440 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T18:45:24.584 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ebtables. 2022-03-10T18:45:24.598 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T18:45:24.616 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T18:45:24.626 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T18:45:24.666 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T18:45:24.683 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T18:45:24.696 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T18:45:24.709 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T18:45:24.746 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T18:45:24.945 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T18:45:24.959 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T18:45:24.995 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T18:45:26.102 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package socat. 2022-03-10T18:45:26.116 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T18:45:26.160 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T18:45:26.182 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package socat. 2022-03-10T18:45:26.194 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T18:45:26.232 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T18:45:26.473 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package socat. 2022-03-10T18:45:26.488 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T18:45:26.523 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T18:45:26.555 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package kubelet. 2022-03-10T18:45:26.567 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T18:45:26.577 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package kubelet. 2022-03-10T18:45:26.589 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T18:45:26.605 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T18:45:26.635 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T18:45:26.909 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package kubelet. 2022-03-10T18:45:26.922 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T18:45:26.984 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T18:45:29.021 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package kubectl. 2022-03-10T18:45:29.024 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package kubectl. 2022-03-10T18:45:29.035 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T18:45:29.038 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T18:45:29.079 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T18:45:29.082 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T18:45:29.403 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package kubectl. 2022-03-10T18:45:29.416 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T18:45:29.453 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T18:45:30.231 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package kubeadm. 2022-03-10T18:45:30.243 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package kubeadm. 2022-03-10T18:45:30.244 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T18:45:30.257 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T18:45:30.281 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T18:45:30.293 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T18:45:30.663 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package kubeadm. 2022-03-10T18:45:30.676 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T18:45:30.713 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T18:45:31.401 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T18:45:31.431 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T18:45:31.518 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T18:45:31.564 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T18:45:31.644 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T18:45:31.698 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T18:45:31.851 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T18:45:31.958 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T18:45:32.084 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T18:45:32.505 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T18:45:32.517 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T18:45:32.626 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T18:45:32.631 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T18:45:32.753 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T18:45:32.757 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T18:45:32.878 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T18:45:32.879 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T18:45:32.883 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T18:45:33.038 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T18:45:33.098 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T18:45:33.106 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T18:45:33.164 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T18:45:33.290 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T18:45:33.503 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T18:45:33.593 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T18:45:33.613 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T18:45:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T18:45:34.124 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T18:45:34.137 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T18:45:34.535 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T18:45:35.440 DEBUG:teuthology.orchestra.run.smithi093:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T18:45:35.443 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T18:45:35.446 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T18:45:48.571 INFO:teuthology.orchestra.run.smithi111.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T18:45:49.155 INFO:teuthology.orchestra.run.smithi093.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T18:45:49.441 INFO:teuthology.orchestra.run.smithi161.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T18:45:51.358 INFO:teuthology.orchestra.run.smithi111.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T18:45:52.049 INFO:teuthology.orchestra.run.smithi093.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T18:45:52.477 INFO:teuthology.orchestra.run.smithi161.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T18:45:53.639 INFO:teuthology.orchestra.run.smithi111.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T18:45:54.321 INFO:teuthology.orchestra.run.smithi093.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T18:45:54.841 INFO:teuthology.orchestra.run.smithi161.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T18:45:57.372 INFO:teuthology.orchestra.run.smithi111.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T18:45:57.987 INFO:teuthology.orchestra.run.smithi093.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T18:45:58.365 INFO:teuthology.orchestra.run.smithi161.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T18:45:58.788 INFO:teuthology.orchestra.run.smithi111.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T18:45:59.554 INFO:teuthology.orchestra.run.smithi093.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T18:45:59.823 INFO:teuthology.orchestra.run.smithi161.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T18:46:04.702 INFO:teuthology.orchestra.run.smithi111.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T18:46:06.439 INFO:teuthology.orchestra.run.smithi161.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T18:46:06.669 INFO:teuthology.orchestra.run.smithi093.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T18:46:06.979 INFO:teuthology.orchestra.run.smithi111.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T18:46:09.230 INFO:teuthology.orchestra.run.smithi161.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T18:46:09.352 INFO:teuthology.orchestra.run.smithi093.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T18:46:09.357 INFO:tasks.kubeadm:Token: abcdef.m8c9wj459ovv1uan 2022-03-10T18:46:09.357 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi093.front.sepia.ceph.com'): '172.21.15.93', Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): '172.21.15.111', Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): '172.21.15.161'} 2022-03-10T18:46:09.358 DEBUG:teuthology.orchestra.run.smithi093:> sudo kubeadm init --node-name smithi093 --token abcdef.m8c9wj459ovv1uan --pod-network-cidr 10.250.224.0/21 2022-03-10T18:46:09.583 INFO:teuthology.orchestra.run.smithi093.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-10T18:46:09.583 INFO:teuthology.orchestra.run.smithi093.stdout:[preflight] Running pre-flight checks 2022-03-10T18:46:10.119 INFO:teuthology.orchestra.run.smithi093.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-10T18:46:10.119 INFO:teuthology.orchestra.run.smithi093.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-10T18:46:10.119 INFO:teuthology.orchestra.run.smithi093.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-10T18:46:10.449 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-10T18:46:10.664 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "ca" certificate and key 2022-03-10T18:46:10.754 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "apiserver" certificate and key 2022-03-10T18:46:10.754 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi093] and IPs [10.96.0.1 172.21.15.93] 2022-03-10T18:46:10.931 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-10T18:46:11.037 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-10T18:46:11.231 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-10T18:46:11.475 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-10T18:46:11.585 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-10T18:46:11.586 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi093] and IPs [172.21.15.93 127.0.0.1 ::1] 2022-03-10T18:46:11.755 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-10T18:46:11.755 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi093] and IPs [172.21.15.93 127.0.0.1 ::1] 2022-03-10T18:46:11.979 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-10T18:46:12.231 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-10T18:46:12.336 INFO:teuthology.orchestra.run.smithi093.stdout:[certs] Generating "sa" key and public key 2022-03-10T18:46:12.336 INFO:teuthology.orchestra.run.smithi093.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-10T18:46:12.577 INFO:teuthology.orchestra.run.smithi093.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-10T18:46:12.796 INFO:teuthology.orchestra.run.smithi093.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-10T18:46:12.897 INFO:teuthology.orchestra.run.smithi093.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-10T18:46:13.248 INFO:teuthology.orchestra.run.smithi093.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-10T18:46:13.260 INFO:teuthology.orchestra.run.smithi093.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T18:46:13.261 INFO:teuthology.orchestra.run.smithi093.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T18:46:13.262 INFO:teuthology.orchestra.run.smithi093.stdout:[kubelet-start] Starting the kubelet 2022-03-10T18:46:13.477 INFO:teuthology.orchestra.run.smithi093.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-10T18:46:13.478 INFO:teuthology.orchestra.run.smithi093.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-10T18:46:13.482 INFO:teuthology.orchestra.run.smithi093.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-10T18:46:13.483 INFO:teuthology.orchestra.run.smithi093.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-10T18:46:13.485 INFO:teuthology.orchestra.run.smithi093.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-10T18:46:13.487 INFO:teuthology.orchestra.run.smithi093.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-10T18:46:30.007 INFO:teuthology.orchestra.run.smithi093.stdout:[apiclient] All control plane components are healthy after 16.519481 seconds 2022-03-10T18:46:30.008 INFO:teuthology.orchestra.run.smithi093.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-10T18:46:30.039 INFO:teuthology.orchestra.run.smithi093.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-10T18:46:30.040 INFO:teuthology.orchestra.run.smithi093.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-10T18:46:30.632 INFO:teuthology.orchestra.run.smithi093.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-10T18:46:30.633 INFO:teuthology.orchestra.run.smithi093.stdout:[mark-control-plane] Marking the node smithi093 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-10T18:46:30.633 INFO:teuthology.orchestra.run.smithi093.stdout:[mark-control-plane] Marking the node smithi093 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-10T18:46:31.171 INFO:teuthology.orchestra.run.smithi093.stdout:[bootstrap-token] Using token: abcdef.m8c9wj459ovv1uan 2022-03-10T18:46:31.171 INFO:teuthology.orchestra.run.smithi093.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-10T18:46:31.223 INFO:teuthology.orchestra.run.smithi093.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-10T18:46:31.265 INFO:teuthology.orchestra.run.smithi093.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-10T18:46:31.273 INFO:teuthology.orchestra.run.smithi093.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-10T18:46:31.281 INFO:teuthology.orchestra.run.smithi093.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-10T18:46:31.290 INFO:teuthology.orchestra.run.smithi093.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-10T18:46:31.431 INFO:teuthology.orchestra.run.smithi093.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-10T18:46:31.928 INFO:teuthology.orchestra.run.smithi093.stdout:[addons] Applied essential addon: CoreDNS 2022-03-10T18:46:32.023 INFO:teuthology.orchestra.run.smithi093.stdout:[addons] Applied essential addon: kube-proxy 2022-03-10T18:46:32.024 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:46:32.024 INFO:teuthology.orchestra.run.smithi093.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-10T18:46:32.024 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:46:32.024 INFO:teuthology.orchestra.run.smithi093.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-10T18:46:32.025 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:46:32.025 INFO:teuthology.orchestra.run.smithi093.stdout: mkdir -p $HOME/.kube 2022-03-10T18:46:32.025 INFO:teuthology.orchestra.run.smithi093.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-10T18:46:32.025 INFO:teuthology.orchestra.run.smithi093.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-10T18:46:32.026 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:46:32.026 INFO:teuthology.orchestra.run.smithi093.stdout:Alternatively, if you are the root user, you can run: 2022-03-10T18:46:32.026 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:46:32.026 INFO:teuthology.orchestra.run.smithi093.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-10T18:46:32.027 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:46:32.027 INFO:teuthology.orchestra.run.smithi093.stdout:You should now deploy a pod network to the cluster. 2022-03-10T18:46:32.027 INFO:teuthology.orchestra.run.smithi093.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-10T18:46:32.027 INFO:teuthology.orchestra.run.smithi093.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-10T18:46:32.028 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:46:32.028 INFO:teuthology.orchestra.run.smithi093.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-10T18:46:32.028 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:46:32.028 INFO:teuthology.orchestra.run.smithi093.stdout:kubeadm join 172.21.15.93:6443 --token abcdef.m8c9wj459ovv1uan \ 2022-03-10T18:46:32.028 INFO:teuthology.orchestra.run.smithi093.stdout: --discovery-token-ca-cert-hash sha256:f71d91ab99a82995d5c2a042f370ac1b01824bc4fbc091ca98d19157af4b1eca 2022-03-10T18:46:32.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo kubeadm join 172.21.15.93:6443 --node-name smithi111 --token abcdef.m8c9wj459ovv1uan --discovery-token-unsafe-skip-ca-verification 2022-03-10T18:46:32.033 DEBUG:teuthology.orchestra.run.smithi161:> sudo kubeadm join 172.21.15.93:6443 --node-name smithi161 --token abcdef.m8c9wj459ovv1uan --discovery-token-unsafe-skip-ca-verification 2022-03-10T18:46:32.073 INFO:teuthology.orchestra.run.smithi111.stdout:[preflight] Running pre-flight checks 2022-03-10T18:46:32.081 INFO:teuthology.orchestra.run.smithi161.stdout:[preflight] Running pre-flight checks 2022-03-10T18:46:45.026 INFO:teuthology.orchestra.run.smithi161.stdout:[preflight] Reading configuration from the cluster... 2022-03-10T18:46:45.026 INFO:teuthology.orchestra.run.smithi161.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-10T18:46:45.027 INFO:teuthology.orchestra.run.smithi111.stdout:[preflight] Reading configuration from the cluster... 2022-03-10T18:46:45.028 INFO:teuthology.orchestra.run.smithi111.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-10T18:46:45.652 INFO:teuthology.orchestra.run.smithi161.stderr:W0310 18:46:45.652625 10749 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-10T18:46:45.653 INFO:teuthology.orchestra.run.smithi111.stderr:W0310 18:46:45.651140 10778 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-10T18:46:45.694 INFO:teuthology.orchestra.run.smithi111.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T18:46:45.694 INFO:teuthology.orchestra.run.smithi111.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T18:46:45.694 INFO:teuthology.orchestra.run.smithi111.stdout:[kubelet-start] Starting the kubelet 2022-03-10T18:46:45.695 INFO:teuthology.orchestra.run.smithi161.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T18:46:45.695 INFO:teuthology.orchestra.run.smithi161.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T18:46:45.695 INFO:teuthology.orchestra.run.smithi161.stdout:[kubelet-start] Starting the kubelet 2022-03-10T18:46:45.893 INFO:teuthology.orchestra.run.smithi111.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-10T18:46:45.927 INFO:teuthology.orchestra.run.smithi161.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-10T18:46:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-03-10T18:46:51.434 INFO:teuthology.orchestra.run.smithi111.stdout:This node has joined the cluster: 2022-03-10T18:46:51.434 INFO:teuthology.orchestra.run.smithi111.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-10T18:46:51.435 INFO:teuthology.orchestra.run.smithi111.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-10T18:46:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-03-10T18:46:51.435 INFO:teuthology.orchestra.run.smithi111.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-10T18:46:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-03-10T18:46:51.461 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-03-10T18:46:51.461 INFO:teuthology.orchestra.run.smithi161.stdout:This node has joined the cluster: 2022-03-10T18:46:51.461 INFO:teuthology.orchestra.run.smithi161.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-10T18:46:51.461 INFO:teuthology.orchestra.run.smithi161.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-10T18:46:51.462 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-03-10T18:46:51.462 INFO:teuthology.orchestra.run.smithi161.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-10T18:46:51.463 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-03-10T18:46:51.466 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:46:51.466 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-10T18:46:51.481 INFO:tasks.kubeadm:Setting up kubectl 2022-03-10T18:46:51.481 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T18:46:51.540 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T18:46:51.557 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T18:46:51.575 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:46:51.576 DEBUG:teuthology.orchestra.run.smithi093:> dd of=.kube/config 2022-03-10T18:46:51.584 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:46:51.584 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/root/.kube/config 2022-03-10T18:46:51.638 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:46:51.639 DEBUG:teuthology.orchestra.run.smithi111:> dd of=.kube/config 2022-03-10T18:46:51.646 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:46:51.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/root/.kube/config 2022-03-10T18:46:51.659 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:46:51.659 DEBUG:teuthology.orchestra.run.smithi161:> dd of=.kube/config 2022-03-10T18:46:51.667 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:46:51.668 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/root/.kube/config 2022-03-10T18:46:51.680 DEBUG:teuthology.orchestra.run.smithi093:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-10T18:46:54.295 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-10T18:46:54.379 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-10T18:46:54.395 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-10T18:46:54.440 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-10T18:46:54.470 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-10T18:46:54.562 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-10T18:46:54.599 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-10T18:46:54.709 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-10T18:46:54.738 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-10T18:46:54.796 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-10T18:46:54.872 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-10T18:46:54.904 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-10T18:46:54.946 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-10T18:46:55.017 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-10T18:46:55.040 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-10T18:46:55.098 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-10T18:46:55.206 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-10T18:46:55.256 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-10T18:46:55.410 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-10T18:46:55.467 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-10T18:46:55.509 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-10T18:46:55.590 INFO:teuthology.orchestra.run.smithi093.stdout:namespace/tigera-operator created 2022-03-10T18:46:55.595 INFO:teuthology.orchestra.run.smithi093.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-10T18:46:55.624 INFO:teuthology.orchestra.run.smithi093.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-10T18:46:55.750 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/tigera-operator created 2022-03-10T18:46:55.758 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-10T18:46:55.787 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-10T18:46:55.822 INFO:teuthology.orchestra.run.smithi093.stdout:deployment.apps/tigera-operator created 2022-03-10T18:46:55.828 DEBUG:teuthology.orchestra.run.smithi093:> kubectl create -f - 2022-03-10T18:46:58.707 INFO:teuthology.orchestra.run.smithi093.stdout:installation.operator.tigera.io/default created 2022-03-10T18:46:58.716 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-10T18:46:58.716 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:46:58.717 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:46:58.723 INFO:tasks.kubeadm:Scratch devices on smithi093: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T18:46:58.724 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T18:46:58.753 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records in 2022-03-10T18:46:58.753 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records out 2022-03-10T18:46:58.754 INFO:teuthology.orchestra.run.smithi093.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0180747 s, 580 MB/s 2022-03-10T18:46:58.755 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T18:46:58.786 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records in 2022-03-10T18:46:58.786 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records out 2022-03-10T18:46:58.787 INFO:teuthology.orchestra.run.smithi093.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.017682 s, 593 MB/s 2022-03-10T18:46:58.788 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T18:46:58.864 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records in 2022-03-10T18:46:58.864 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records out 2022-03-10T18:46:58.864 INFO:teuthology.orchestra.run.smithi093.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0190924 s, 549 MB/s 2022-03-10T18:46:58.866 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T18:46:58.892 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records in 2022-03-10T18:46:58.892 INFO:teuthology.orchestra.run.smithi093.stderr:10+0 records out 2022-03-10T18:46:58.892 INFO:teuthology.orchestra.run.smithi093.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0164823 s, 636 MB/s 2022-03-10T18:46:58.894 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:46:58.894 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:46:58.900 INFO:tasks.kubeadm:Scratch devices on smithi111: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T18:46:58.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T18:46:58.970 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records in 2022-03-10T18:46:58.971 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2022-03-10T18:46:58.971 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01753 s, 598 MB/s 2022-03-10T18:46:58.973 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T18:46:59.006 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records in 2022-03-10T18:46:59.007 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2022-03-10T18:46:59.007 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0178284 s, 588 MB/s 2022-03-10T18:46:59.010 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T18:46:59.089 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records in 2022-03-10T18:46:59.090 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2022-03-10T18:46:59.090 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0193362 s, 542 MB/s 2022-03-10T18:46:59.091 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T18:46:59.121 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records in 2022-03-10T18:46:59.122 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2022-03-10T18:46:59.122 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0158797 s, 660 MB/s 2022-03-10T18:46:59.123 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:46:59.124 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:46:59.132 INFO:tasks.kubeadm:Scratch devices on smithi161: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T18:46:59.133 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T18:46:59.160 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records in 2022-03-10T18:46:59.160 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2022-03-10T18:46:59.160 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0160791 s, 652 MB/s 2022-03-10T18:46:59.161 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T18:46:59.189 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records in 2022-03-10T18:46:59.189 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2022-03-10T18:46:59.189 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150653 s, 696 MB/s 2022-03-10T18:46:59.191 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T18:46:59.264 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records in 2022-03-10T18:46:59.264 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2022-03-10T18:46:59.264 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163904 s, 640 MB/s 2022-03-10T18:46:59.265 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T18:46:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records in 2022-03-10T18:46:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2022-03-10T18:46:59.293 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0157387 s, 666 MB/s 2022-03-10T18:46:59.313 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-10T18:46:59.313 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi093-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi093 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi093-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi093 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi093-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi093 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi093-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi093 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi111-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi111 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi111-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi111 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi111-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi111 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi111-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi111 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi161-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi161 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi161-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi161 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi161-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi161 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi161-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi161 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-10T18:46:59.313 DEBUG:teuthology.orchestra.run.smithi093:> kubectl create -f - 2022-03-10T18:47:00.199 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi093-lv-1 created 2022-03-10T18:47:00.232 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi093-lv-2 created 2022-03-10T18:47:00.236 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi093-lv-3 created 2022-03-10T18:47:00.269 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi093-lv-4 created 2022-03-10T18:47:00.279 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi111-lv-1 created 2022-03-10T18:47:00.287 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi111-lv-2 created 2022-03-10T18:47:00.311 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi111-lv-3 created 2022-03-10T18:47:00.320 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi111-lv-4 created 2022-03-10T18:47:00.366 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi161-lv-1 created 2022-03-10T18:47:00.379 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi161-lv-2 created 2022-03-10T18:47:00.403 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi161-lv-3 created 2022-03-10T18:47:00.412 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi161-lv-4 created 2022-03-10T18:47:00.437 INFO:teuthology.orchestra.run.smithi093.stdout:storageclass.storage.k8s.io/scratch created 2022-03-10T18:47:00.443 DEBUG:teuthology.orchestra.run.smithi093:> kubectl taint node smithi093 node-role.kubernetes.io/master- || true 2022-03-10T18:47:00.507 INFO:teuthology.orchestra.run.smithi093.stdout:node/smithi093 untainted 2022-03-10T18:47:00.510 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-10T18:47:00.510 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-10T18:47:00.524 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-10T18:47:00.525 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:47:00.525 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:47:00.533 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T18:47:00.533 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T18:47:00.540 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T18:47:00.541 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:00.541 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 5h/5d Inode: 501 Links: 1 2022-03-10T18:47:00.541 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:00.541 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2022-03-10 18:46:58.751029628 +0000 2022-03-10T18:47:00.542 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2022-03-10 18:46:58.751029628 +0000 2022-03-10T18:47:00.542 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2022-03-10 18:46:58.751029628 +0000 2022-03-10T18:47:00.543 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: - 2022-03-10T18:47:00.549 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2022-03-10T18:47:00.550 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2022-03-10T18:47:00.550 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000597342 s, 857 kB/s 2022-03-10T18:47:00.554 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T18:47:00.604 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T18:47:00.605 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:00.605 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-10T18:47:00.605 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:00.605 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2022-03-10 18:46:58.783029627 +0000 2022-03-10T18:47:00.606 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2022-03-10 18:46:58.783029627 +0000 2022-03-10T18:47:00.606 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2022-03-10 18:46:58.783029627 +0000 2022-03-10T18:47:00.606 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: - 2022-03-10T18:47:00.614 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2022-03-10T18:47:00.615 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2022-03-10T18:47:00.615 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000549192 s, 932 kB/s 2022-03-10T18:47:00.619 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T18:47:00.623 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T18:47:00.624 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:00.624 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-03-10T18:47:00.624 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:00.624 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2022-03-10 18:46:58.863029625 +0000 2022-03-10T18:47:00.625 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2022-03-10 18:46:58.863029625 +0000 2022-03-10T18:47:00.625 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2022-03-10 18:46:58.863029625 +0000 2022-03-10T18:47:00.625 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: - 2022-03-10T18:47:00.631 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2022-03-10T18:47:00.631 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2022-03-10T18:47:00.632 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000491178 s, 1.0 MB/s 2022-03-10T18:47:00.635 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T18:47:00.641 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T18:47:00.641 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:00.642 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-10T18:47:00.642 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:00.642 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2022-03-10 18:46:58.891029624 +0000 2022-03-10T18:47:00.642 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2022-03-10 18:46:58.891029624 +0000 2022-03-10T18:47:00.643 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2022-03-10 18:46:58.891029624 +0000 2022-03-10T18:47:00.643 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: - 2022-03-10T18:47:00.650 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2022-03-10T18:47:00.650 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2022-03-10T18:47:00.650 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000532487 s, 962 kB/s 2022-03-10T18:47:00.655 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T18:47:00.763 INFO:teuthology.orchestra.run.smithi093.stdout:loop 2022-03-10T18:47:00.765 INFO:tasks.nvme_loop:Connecting nvme_loop smithi093:/dev/vg_nvme/lv_1... 2022-03-10T18:47:00.765 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T18:47:00.786 INFO:teuthology.orchestra.run.smithi093.stdout:1 2022-03-10T18:47:00.810 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/vg_nvme/lv_11 2022-03-10T18:47:00.835 INFO:tasks.nvme_loop:Connecting nvme_loop smithi093:/dev/vg_nvme/lv_2... 2022-03-10T18:47:00.931 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T18:47:00.954 INFO:teuthology.orchestra.run.smithi093.stdout:1 2022-03-10T18:47:00.975 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/vg_nvme/lv_21 2022-03-10T18:47:00.996 INFO:tasks.nvme_loop:Connecting nvme_loop smithi093:/dev/vg_nvme/lv_3... 2022-03-10T18:47:00.996 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T18:47:01.060 INFO:teuthology.orchestra.run.smithi093.stdout:1 2022-03-10T18:47:01.083 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/vg_nvme/lv_31 2022-03-10T18:47:01.105 INFO:tasks.nvme_loop:Connecting nvme_loop smithi093:/dev/vg_nvme/lv_4... 2022-03-10T18:47:01.106 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T18:47:01.127 INFO:teuthology.orchestra.run.smithi093.stdout:1 2022-03-10T18:47:01.159 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/vg_nvme/lv_41 2022-03-10T18:47:01.182 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:47:01.183 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:47:01.190 DEBUG:teuthology.orchestra.run.smithi093:> sudo nvme list 2022-03-10T18:47:01.249 INFO:teuthology.orchestra.run.smithi093.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T18:47:01.250 INFO:teuthology.orchestra.run.smithi093.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T18:47:01.250 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/nvme0n1 PHFT62040150400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-10T18:47:01.251 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/nvme1n1 3f8c87c7ff3c42fd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:01.251 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/nvme2n1 adce2a500f0d20cf Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:01.251 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/nvme3n1 9a4cd88dbf761f1c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:01.251 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/nvme4n1 62d4a3de0f12b3d7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:01.253 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T18:47:01.253 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:47:01.253 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/scratch_devs 2022-03-10T18:47:01.266 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:47:01.267 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:47:01.275 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T18:47:01.276 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T18:47:01.326 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T18:47:01.327 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:01.327 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-10T18:47:01.327 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:01.328 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2022-03-10 18:46:58.969800708 +0000 2022-03-10T18:47:01.328 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2022-03-10 18:46:58.969800708 +0000 2022-03-10T18:47:01.328 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2022-03-10 18:46:58.969800708 +0000 2022-03-10T18:47:01.328 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2022-03-10T18:47:01.335 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2022-03-10T18:47:01.336 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2022-03-10T18:47:01.336 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000550285 s, 930 kB/s 2022-03-10T18:47:01.340 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T18:47:01.346 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T18:47:01.347 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:01.347 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-03-10T18:47:01.347 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:01.348 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2022-03-10 18:46:59.009800707 +0000 2022-03-10T18:47:01.348 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2022-03-10 18:46:59.005800707 +0000 2022-03-10T18:47:01.348 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2022-03-10 18:46:59.005800707 +0000 2022-03-10T18:47:01.348 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2022-03-10T18:47:01.353 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2022-03-10T18:47:01.353 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2022-03-10T18:47:01.354 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000562698 s, 910 kB/s 2022-03-10T18:47:01.357 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T18:47:01.362 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T18:47:01.363 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:01.363 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-10T18:47:01.363 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:01.363 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2022-03-10 18:46:59.089800705 +0000 2022-03-10T18:47:01.364 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2022-03-10 18:46:59.089800705 +0000 2022-03-10T18:47:01.364 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2022-03-10 18:46:59.089800705 +0000 2022-03-10T18:47:01.365 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2022-03-10T18:47:01.371 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2022-03-10T18:47:01.372 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2022-03-10T18:47:01.372 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000507955 s, 1.0 MB/s 2022-03-10T18:47:01.376 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T18:47:01.381 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T18:47:01.381 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:01.382 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-10T18:47:01.382 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:01.382 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2022-03-10 18:46:59.121800705 +0000 2022-03-10T18:47:01.382 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2022-03-10 18:46:59.121800705 +0000 2022-03-10T18:47:01.383 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2022-03-10 18:46:59.121800705 +0000 2022-03-10T18:47:01.383 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2022-03-10T18:47:01.390 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2022-03-10T18:47:01.391 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2022-03-10T18:47:01.391 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000569946 s, 898 kB/s 2022-03-10T18:47:01.394 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T18:47:01.481 INFO:teuthology.orchestra.run.smithi111.stdout:loop 2022-03-10T18:47:01.482 INFO:tasks.nvme_loop:Connecting nvme_loop smithi111:/dev/vg_nvme/lv_1... 2022-03-10T18:47:01.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T18:47:01.501 INFO:teuthology.orchestra.run.smithi111.stdout:1 2022-03-10T18:47:01.521 INFO:teuthology.orchestra.run.smithi111.stdout:/dev/vg_nvme/lv_11 2022-03-10T18:47:01.539 INFO:tasks.nvme_loop:Connecting nvme_loop smithi111:/dev/vg_nvme/lv_2... 2022-03-10T18:47:01.540 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T18:47:01.557 INFO:teuthology.orchestra.run.smithi111.stdout:1 2022-03-10T18:47:01.578 INFO:teuthology.orchestra.run.smithi111.stdout:/dev/vg_nvme/lv_21 2022-03-10T18:47:01.596 INFO:tasks.nvme_loop:Connecting nvme_loop smithi111:/dev/vg_nvme/lv_3... 2022-03-10T18:47:01.596 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T18:47:01.613 INFO:teuthology.orchestra.run.smithi111.stdout:1 2022-03-10T18:47:01.630 INFO:teuthology.orchestra.run.smithi111.stdout:/dev/vg_nvme/lv_31 2022-03-10T18:47:01.645 INFO:tasks.nvme_loop:Connecting nvme_loop smithi111:/dev/vg_nvme/lv_4... 2022-03-10T18:47:01.645 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T18:47:01.664 INFO:teuthology.orchestra.run.smithi111.stdout:1 2022-03-10T18:47:01.689 INFO:teuthology.orchestra.run.smithi111.stdout:/dev/vg_nvme/lv_41 2022-03-10T18:47:01.708 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:47:01.709 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:47:01.757 DEBUG:teuthology.orchestra.run.smithi111:> sudo nvme list 2022-03-10T18:47:01.768 INFO:teuthology.orchestra.run.smithi111.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T18:47:01.768 INFO:teuthology.orchestra.run.smithi111.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T18:47:01.768 INFO:teuthology.orchestra.run.smithi111.stdout:/dev/nvme0n1 PHFT620400VZ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-10T18:47:01.769 INFO:teuthology.orchestra.run.smithi111.stdout:/dev/nvme1n1 931a4471b8f13ad7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:01.769 INFO:teuthology.orchestra.run.smithi111.stdout:/dev/nvme2n1 5080d05f7892cdef Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:01.769 INFO:teuthology.orchestra.run.smithi111.stdout:/dev/nvme3n1 bf3ed115818999ba Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:01.769 INFO:teuthology.orchestra.run.smithi111.stdout:/dev/nvme4n1 b27275bc64267d6e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:01.770 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T18:47:01.771 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:47:01.771 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/scratch_devs 2022-03-10T18:47:01.785 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:47:01.785 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:47:01.793 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T18:47:01.794 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T18:47:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T18:47:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:01.847 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-10T18:47:01.848 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:01.849 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-03-10 18:46:59.163048427 +0000 2022-03-10T18:47:01.849 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-03-10 18:46:59.163048427 +0000 2022-03-10T18:47:01.849 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-03-10 18:46:59.163048427 +0000 2022-03-10T18:47:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-03-10T18:47:01.857 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-03-10T18:47:01.858 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-03-10T18:47:01.858 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000605929 s, 845 kB/s 2022-03-10T18:47:01.862 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T18:47:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T18:47:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:01.870 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-10T18:47:01.870 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:01.870 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-03-10 18:46:59.195048426 +0000 2022-03-10T18:47:01.870 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-03-10 18:46:59.195048426 +0000 2022-03-10T18:47:01.871 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-03-10 18:46:59.195048426 +0000 2022-03-10T18:47:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-03-10T18:47:01.877 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-03-10T18:47:01.877 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-03-10T18:47:01.878 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000559804 s, 915 kB/s 2022-03-10T18:47:01.881 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T18:47:01.930 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T18:47:01.930 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:01.930 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-03-10T18:47:01.931 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:01.931 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-03-10 18:46:59.267048424 +0000 2022-03-10T18:47:01.931 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-03-10 18:46:59.267048424 +0000 2022-03-10T18:47:01.931 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-03-10 18:46:59.267048424 +0000 2022-03-10T18:47:01.932 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-03-10T18:47:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-03-10T18:47:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-03-10T18:47:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.00053984 s, 948 kB/s 2022-03-10T18:47:01.943 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T18:47:01.949 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T18:47:01.950 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T18:47:01.950 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-03-10T18:47:01.950 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T18:47:01.950 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-03-10 18:46:59.295048424 +0000 2022-03-10T18:47:01.951 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-03-10 18:46:59.295048424 +0000 2022-03-10T18:47:01.951 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-03-10 18:46:59.295048424 +0000 2022-03-10T18:47:01.951 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-03-10T18:47:01.956 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-03-10T18:47:01.957 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-03-10T18:47:01.957 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000538475 s, 951 kB/s 2022-03-10T18:47:01.961 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T18:47:02.034 INFO:teuthology.orchestra.run.smithi161.stdout:loop 2022-03-10T18:47:02.035 INFO:tasks.nvme_loop:Connecting nvme_loop smithi161:/dev/vg_nvme/lv_1... 2022-03-10T18:47:02.036 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T18:47:02.060 INFO:teuthology.orchestra.run.smithi161.stdout:1 2022-03-10T18:47:02.086 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/vg_nvme/lv_11 2022-03-10T18:47:02.110 INFO:tasks.nvme_loop:Connecting nvme_loop smithi161:/dev/vg_nvme/lv_2... 2022-03-10T18:47:02.111 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T18:47:02.134 INFO:teuthology.orchestra.run.smithi161.stdout:1 2022-03-10T18:47:02.159 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/vg_nvme/lv_21 2022-03-10T18:47:02.182 INFO:tasks.nvme_loop:Connecting nvme_loop smithi161:/dev/vg_nvme/lv_3... 2022-03-10T18:47:02.182 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T18:47:02.202 INFO:teuthology.orchestra.run.smithi161.stdout:1 2022-03-10T18:47:02.224 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/vg_nvme/lv_31 2022-03-10T18:47:02.245 INFO:tasks.nvme_loop:Connecting nvme_loop smithi161:/dev/vg_nvme/lv_4... 2022-03-10T18:47:02.245 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T18:47:02.312 INFO:teuthology.orchestra.run.smithi161.stdout:1 2022-03-10T18:47:02.337 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/vg_nvme/lv_41 2022-03-10T18:47:02.362 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:47:02.363 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:47:02.369 DEBUG:teuthology.orchestra.run.smithi161:> sudo nvme list 2022-03-10T18:47:02.381 INFO:teuthology.orchestra.run.smithi161.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T18:47:02.381 INFO:teuthology.orchestra.run.smithi161.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T18:47:02.382 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/nvme0n1 CVFT6233001X400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-10T18:47:02.382 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/nvme1n1 4d3f513909713180 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:02.382 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/nvme2n1 aa199e46c2e9d6d6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:02.382 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/nvme3n1 f622aa9d72aa91d0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:02.384 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-10T18:47:03.385 DEBUG:teuthology.orchestra.run.smithi161:> sudo nvme list 2022-03-10T18:47:03.402 INFO:teuthology.orchestra.run.smithi161.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T18:47:03.402 INFO:teuthology.orchestra.run.smithi161.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T18:47:03.402 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/nvme0n1 CVFT6233001X400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-10T18:47:03.403 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/nvme1n1 4d3f513909713180 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:03.403 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/nvme2n1 aa199e46c2e9d6d6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:03.403 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/nvme3n1 f622aa9d72aa91d0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:03.403 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/nvme4n1 43c39d0d7c4ece41 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T18:47:03.404 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T18:47:03.405 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:47:03.405 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/scratch_devs 2022-03-10T18:47:03.422 INFO:teuthology.run_tasks:Running task rook... 2022-03-10T18:47:03.495 INFO:tasks.rook:Rook start 2022-03-10T18:47:03.496 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '78e1eb94d97cd01be975676a7cefdde0c403b210', 'spec': {'mon': {'allowMultiplePerNode': False}, 'network': {'provider': 'host'}}, 'rook_image': 'rook/ceph:master'} 2022-03-10T18:47:03.496 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210 2022-03-10T18:47:03.497 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-10T18:47:03.497 DEBUG:teuthology.orchestra.run.smithi093:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-10T18:47:03.507 INFO:teuthology.orchestra.run.smithi093.stderr:Cloning into 'rook'... 2022-03-10T18:47:07.549 INFO:tasks.rook:/home/teuthworker 2022-03-10T18:47:07.549 DEBUG:teuthology.orchestra.run.smithi093:> PATH=/sbin:/usr/sbin ip addr show 2022-03-10T18:47:07.597 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-10T18:47:07.598 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-10T18:47:07.598 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2022-03-10T18:47:07.598 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2022-03-10T18:47:07.598 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2022-03-10T18:47:07.599 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2022-03-10T18:47:07.599 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T18:47:07.599 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2022-03-10T18:47:07.599 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2022-03-10T18:47:07.600 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T18:47:07.600 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2022-03-10T18:47:07.600 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2022-03-10T18:47:07.600 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T18:47:07.600 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2022-03-10T18:47:07.601 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-10T18:47:07.601 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2022-03-10T18:47:07.601 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-10T18:47:07.601 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 42847sec preferred_lft 42847sec 2022-03-10T18:47:07.601 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2022-03-10T18:47:07.602 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2022-03-10T18:47:07.602 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-10T18:47:07.602 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:1f:07:8b:af brd ff:ff:ff:ff:ff:ff 2022-03-10T18:47:07.602 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-10T18:47:07.603 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2022-03-10T18:47:07.604 DEBUG:teuthology.orchestra.run.smithi093:> which systemctl 2022-03-10T18:47:07.611 INFO:teuthology.orchestra.run.smithi093.stdout:/usr/bin/systemctl 2022-03-10T18:47:07.612 DEBUG:teuthology.orchestra.run.smithi093:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-10T18:47:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T18:47:07.616 DEBUG:teuthology.orchestra.run.smithi093:> true 2022-03-10T18:47:07.646 DEBUG:teuthology.orchestra.run.smithi093:> sudo lsb_release -is 2022-03-10T18:47:07.696 INFO:teuthology.orchestra.run.smithi093.stdout:Ubuntu 2022-03-10T18:47:07.701 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-10T18:47:07.701 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-10T18:47:07.711 DEBUG:teuthology.orchestra.remote:smithi093:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T18:47:07.726 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:47:07.727 DEBUG:teuthology.orchestra.run.smithi093:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-10T18:47:07.738 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-10T18:47:07.780 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:47:07.781 DEBUG:teuthology.orchestra.run.smithi093:> dd of=operator.yaml 2022-03-10T18:47:07.786 INFO:tasks.rook:Deploying operator 2022-03-10T18:47:07.795 DEBUG:teuthology.orchestra.remote:smithi093:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T18:47:07.815 DEBUG:teuthology.orchestra.remote:smithi093:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T18:47:07.829 DEBUG:teuthology.orchestra.run.smithi093:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-10T18:47:08.129 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-10T18:47:08.153 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-10T18:47:08.179 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-10T18:47:08.261 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-10T18:47:08.376 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-10T18:47:08.418 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-10T18:47:08.452 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-10T18:47:08.479 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-10T18:47:08.500 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-10T18:47:08.513 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-10T18:47:08.595 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-10T18:47:08.601 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-10T18:47:08.628 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-10T18:47:08.662 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-10T18:47:08.762 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-10T18:47:08.795 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-10T18:47:08.845 INFO:teuthology.orchestra.run.smithi093.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-10T18:47:08.944 INFO:teuthology.orchestra.run.smithi093.stdout:namespace/rook-ceph created 2022-03-10T18:47:08.950 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-10T18:47:08.979 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-10T18:47:09.006 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-10T18:47:09.028 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T18:47:09.121 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-10T18:47:09.126 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-10T18:47:09.145 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-10T18:47:09.179 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-10T18:47:09.185 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-10T18:47:09.212 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-10T18:47:09.260 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T18:47:09.263 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T18:47:09.268 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-10T18:47:09.276 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-10T18:47:09.285 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T18:47:09.293 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-10T18:47:09.321 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-10T18:47:09.326 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-10T18:47:09.334 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-10T18:47:09.343 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T18:47:09.351 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T18:47:09.414 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-10T18:47:09.418 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-10T18:47:09.426 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-10T18:47:09.435 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-10T18:47:09.443 INFO:teuthology.orchestra.run.smithi093.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-10T18:47:09.451 INFO:teuthology.orchestra.run.smithi093.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-10T18:47:09.452 INFO:teuthology.orchestra.run.smithi093.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-10T18:47:09.460 INFO:teuthology.orchestra.run.smithi093.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-10T18:47:09.491 INFO:teuthology.orchestra.run.smithi093.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T18:47:09.493 INFO:teuthology.orchestra.run.smithi093.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-10T18:47:09.501 INFO:teuthology.orchestra.run.smithi093.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-10T18:47:09.510 INFO:teuthology.orchestra.run.smithi093.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-10T18:47:09.565 INFO:teuthology.orchestra.run.smithi093.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T18:47:09.568 INFO:teuthology.orchestra.run.smithi093.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-10T18:47:09.596 INFO:teuthology.orchestra.run.smithi093.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T18:47:09.601 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-10T18:47:09.611 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-10T18:47:09.618 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-10T18:47:09.626 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-10T18:47:09.635 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-10T18:47:09.663 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-10T18:47:09.714 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-10T18:47:09.717 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-10T18:47:09.726 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-10T18:47:09.735 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-10T18:47:09.743 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T18:47:09.751 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-10T18:47:09.780 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-10T18:47:09.784 INFO:teuthology.orchestra.run.smithi093.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T18:47:09.793 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-10T18:47:09.801 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/rook-ceph-mgr created 2022-03-10T18:47:09.810 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/rook-ceph-osd created 2022-03-10T18:47:09.907 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-10T18:47:09.918 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/rook-ceph-system created 2022-03-10T18:47:09.947 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-10T18:47:09.971 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-10T18:47:09.999 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-10T18:47:10.063 INFO:teuthology.orchestra.run.smithi093.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-10T18:47:10.089 INFO:teuthology.orchestra.run.smithi093.stdout:configmap/rook-ceph-operator-config created 2022-03-10T18:47:10.106 INFO:teuthology.orchestra.run.smithi093.stdout:deployment.apps/rook-ceph-operator created 2022-03-10T18:47:10.110 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T18:47:10.233 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:47:10.233 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-operator-75dd789779-98vhg 0/1 Pending 0 0s 2022-03-10T18:47:20.238 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T18:47:20.340 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:47:20.340 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-operator-75dd789779-98vhg 0/1 Pending 0 10s 2022-03-10T18:47:30.343 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T18:47:30.410 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:47:30.410 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-operator-75dd789779-98vhg 0/1 Pending 0 20s 2022-03-10T18:47:40.412 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T18:47:40.604 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:47:40.604 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-operator-75dd789779-98vhg 0/1 ContainerCreating 0 30s 2022-03-10T18:47:50.608 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T18:47:50.668 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:47:50.668 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-operator-75dd789779-98vhg 0/1 ContainerCreating 0 40s 2022-03-10T18:48:00.671 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T18:48:00.738 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:48:00.738 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-operator-75dd789779-98vhg 0/1 ContainerCreating 0 50s 2022-03-10T18:48:10.742 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T18:48:11.019 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:48:11.019 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-operator-75dd789779-98vhg 0/1 ContainerCreating 0 61s 2022-03-10T18:48:21.021 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T18:48:21.083 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:48:21.084 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-operator-75dd789779-98vhg 0/1 ContainerCreating 0 71s 2022-03-10T18:48:31.093 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T18:48:31.153 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:48:31.154 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-operator-75dd789779-98vhg 1/1 Running 0 81s 2022-03-10T18:48:31.156 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-98vhg 2022-03-10T18:48:31.164 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:48:31.165 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:48:31.171 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:48:31.172 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:48:31.180 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:48:31.180 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T18:48:31.190 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-10T18:48:31.190 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-10T18:48:31.191 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-10T18:48:31.191 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-10T18:48:31.191 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-10T18:48:31.192 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-10T18:48:31.192 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-10T18:48:31.192 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-10T18:48:31.195 DEBUG:teuthology.orchestra.run.smithi093:> kubectl create -f - 2022-03-10T18:48:31.250 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:22.880852 I | rookcmd: starting Rook v1.8.0-alpha.0.399.g5294678f0 with arguments '/usr/local/bin/rook ceph operator' 2022-03-10T18:48:31.251 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:22.880965 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-10T18:48:31.251 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:22.880971 I | cephcmd: starting Rook-Ceph operator 2022-03-10T18:48:31.251 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.033508 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-10T18:48:31.252 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.040448 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T18:48:31.252 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.040469 I | operator: watching all namespaces for Ceph CRs 2022-03-10T18:48:31.252 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.040507 I | operator: setting up schemes 2022-03-10T18:48:31.252 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.043034 I | operator: setting up the controller-runtime manager 2022-03-10T18:48:31.252 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.350199 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T18:48:31.253 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353131 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T18:48:31.253 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353174 I | ceph-cluster-controller: successfully started 2022-03-10T18:48:31.253 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353245 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T18:48:31.253 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353269 I | ceph-crashcollector-controller: successfully started 2022-03-10T18:48:31.254 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353289 I | ceph-block-pool-controller: successfully started 2022-03-10T18:48:31.254 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353318 I | ceph-object-store-user-controller: successfully started 2022-03-10T18:48:31.254 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353345 I | ceph-object-realm-controller: successfully started 2022-03-10T18:48:31.254 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353359 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T18:48:31.255 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353372 I | ceph-object-zone-controller: successfully started 2022-03-10T18:48:31.255 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353485 I | ceph-object-controller: successfully started 2022-03-10T18:48:31.255 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353548 I | ceph-file-controller: successfully started 2022-03-10T18:48:31.255 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353584 I | ceph-nfs-controller: successfully started 2022-03-10T18:48:31.255 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353612 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T18:48:31.256 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353638 I | ceph-client-controller: successfully started 2022-03-10T18:48:31.256 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353658 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T18:48:31.256 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353718 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T18:48:31.256 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353746 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T18:48:31.257 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353762 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T18:48:31.257 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353805 I | ceph-bucket-topic: successfully started 2022-03-10T18:48:31.257 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353817 I | ceph-bucket-notification: successfully started 2022-03-10T18:48:31.257 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353826 I | ceph-bucket-notification: successfully started 2022-03-10T18:48:31.257 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.353835 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T18:48:31.258 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.354707 I | operator: starting the controller-runtime manager 2022-03-10T18:48:31.258 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.457704 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T18:48:31.258 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.457743 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T18:48:31.258 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.457759 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T18:48:31.259 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.461402 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-10T18:48:31.259 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.461446 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-10T18:48:31.259 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.462065 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T18:48:31.259 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.464100 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-10T18:48:31.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.469979 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-10T18:48:31.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.469996 I | ceph-csi: CSI CephFS driver disabled 2022-03-10T18:48:31.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.470012 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-10T18:48:31.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.471606 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-10T18:48:31.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:23.476582 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-10T18:48:32.183 INFO:teuthology.orchestra.run.smithi093.stdout:configmap/rook-config-override created 2022-03-10T18:48:32.189 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: false count: 3 network: provider: host skipUpgradeChecks: true 2022-03-10T18:48:32.190 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:48:32.190 DEBUG:teuthology.orchestra.run.smithi093:> dd of=cluster.yaml 2022-03-10T18:48:32.195 DEBUG:teuthology.orchestra.run.smithi093:> kubectl create -f cluster.yaml 2022-03-10T18:48:34.570 INFO:teuthology.orchestra.run.smithi093.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-10T18:48:34.578 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.576379 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-10T18:48:34.586 DEBUG:teuthology.orchestra.remote:smithi093:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T18:48:34.629 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.629142 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T18:48:34.635 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.631536 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-10T18:48:34.639 DEBUG:teuthology.orchestra.run.smithi093:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-10T18:48:34.640 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.639866 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T18:48:34.666 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662607 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T18:48:34.667 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662632 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T18:48:34.667 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662639 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T18:48:34.667 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662644 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T18:48:34.668 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662649 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T18:48:34.668 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662654 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T18:48:34.668 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662661 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T18:48:34.668 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662670 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T18:48:34.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662678 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T18:48:34.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662685 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T18:48:34.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662692 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T18:48:34.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662700 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T18:48:34.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662708 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T18:48:34.670 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662715 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T18:48:34.670 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662722 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T18:48:34.670 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662730 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T18:48:34.670 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662869 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:48:34.671 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.662942 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:48:34.671 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:48:34.664257 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210... 2022-03-10T18:48:34.901 INFO:teuthology.orchestra.run.smithi093.stdout:deployment.apps/rook-ceph-tools created 2022-03-10T18:48:34.906 INFO:tasks.rook:Waiting for tools container to start 2022-03-10T18:48:34.906 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:48:35.017 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:48:35.019 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 Pending 0 1s 2022-03-10T18:48:40.024 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:48:40.093 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:48:40.094 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 6s 2022-03-10T18:48:45.096 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:48:45.185 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:48:45.186 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 11s 2022-03-10T18:48:50.189 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:48:50.252 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:48:50.252 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 16s 2022-03-10T18:48:55.255 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:48:55.319 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:48:55.320 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 21s 2022-03-10T18:49:00.322 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:00.391 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:00.391 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 26s 2022-03-10T18:49:05.394 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:05.458 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:05.459 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 31s 2022-03-10T18:49:10.462 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:10.522 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:10.523 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 36s 2022-03-10T18:49:15.526 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:15.605 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:15.606 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 41s 2022-03-10T18:49:20.608 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:20.676 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:20.676 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 46s 2022-03-10T18:49:25.679 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:25.748 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:25.749 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 51s 2022-03-10T18:49:30.751 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:30.824 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:30.825 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 56s 2022-03-10T18:49:35.827 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:35.891 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:35.892 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 61s 2022-03-10T18:49:40.895 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:40.965 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:40.965 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 66s 2022-03-10T18:49:45.969 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:46.041 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:46.041 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 72s 2022-03-10T18:49:51.045 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:51.112 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:51.112 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 77s 2022-03-10T18:49:56.114 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:49:56.173 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:49:56.174 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 82s 2022-03-10T18:50:01.179 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:01.251 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:01.251 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 87s 2022-03-10T18:50:06.255 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:06.318 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:06.319 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 92s 2022-03-10T18:50:11.321 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:11.391 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:11.391 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 97s 2022-03-10T18:50:11.734 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734742 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T18:50:11.734 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734791 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T18:50:11.735 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734811 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T18:50:11.735 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734821 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T18:50:11.735 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734833 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T18:50:11.735 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734843 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T18:50:11.738 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734855 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T18:50:11.738 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734866 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T18:50:11.738 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734877 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T18:50:11.739 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734888 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T18:50:11.739 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734899 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T18:50:11.739 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734910 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T18:50:11.739 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734920 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T18:50:11.740 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734930 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T18:50:11.740 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734940 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T18:50:11.740 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734952 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T18:50:11.740 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734963 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T18:50:11.741 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734974 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T18:50:11.741 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734983 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T18:50:11.741 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.734994 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T18:50:11.741 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.735004 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T18:50:11.741 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.735015 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T18:50:11.742 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.735025 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T18:50:11.742 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.735038 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T18:50:11.742 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.735078 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T18:50:11.743 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.744565 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T18:50:11.751 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.752080 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:50:11.752 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.752109 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:50:11.752 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.752114 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:50:11.753 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.752116 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:50:11.753 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.752119 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:50:11.753 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.752121 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:50:11.754 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.752124 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T18:50:11.815 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.816117 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:50:11.815 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.816150 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:50:11.815 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.816158 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T18:50:11.900 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:11.901440 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T18:50:12.002 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:12.001284 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:50:12.002 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:12.001317 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:50:12.002 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:12.001325 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T18:50:12.135 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:12.136343 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:50:12.136 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:12.136476 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:50:12.136 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:12.136562 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T18:50:12.189 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:12.190166 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T18:50:12.821 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:12.822345 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T18:50:13.364 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:13.363671 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T18:50:13.365 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:13.363709 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T18:50:13.683 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:13.684175 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T18:50:16.394 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:16.450 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:16.451 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 102s 2022-03-10T18:50:21.454 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:21.519 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:21.520 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 107s 2022-03-10T18:50:22.000 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.001423 I | ceph-spec: detected ceph image version: "17.0.0-11084 quincy" 2022-03-10T18:50:22.001 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.001501 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T18:50:22.001 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.001526 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11084 quincy", pursuing 2022-03-10T18:50:22.011 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.012467 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11084 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210" 2022-03-10T18:50:22.062 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.063400 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-10 18:50:22.043396059 +0000 UTC m=+119.186338801 LastTransitionTime:2022-03-10 18:50:22.043395899 +0000 UTC m=+119.186338675}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:50:22.076 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.077673 I | op-mon: start running mons 2022-03-10T18:50:22.151 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.152555 I | op-mon: creating mon secrets for a new cluster 2022-03-10T18:50:22.176 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.177788 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-10T18:50:22.179 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.180775 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-10T18:50:22.911 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.912814 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:50:22.911 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:22.912963 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:50:24.513 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:24.514828 I | op-mon: targeting the mon count 3 2022-03-10T18:50:24.520 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:24.521766 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-10T18:50:24.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:24.545747 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-10T18:50:24.571 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:24.573025 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-10T18:50:25.912 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:25.913558 I | op-mon: parsing mon endpoints: 2022-03-10T18:50:25.913 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:25.913675 W | op-mon: ignoring invalid monitor 2022-03-10T18:50:25.913 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:25.913773 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T18:50:25.913 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:25.913807 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T18:50:25.916 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:25.915097 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi161 2022-03-10T18:50:25.916 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:25.915118 I | op-mon: mon a assigned to node smithi161 2022-03-10T18:50:25.916 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:25.915328 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi093 2022-03-10T18:50:25.916 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:25.915347 I | op-mon: mon b assigned to node smithi093 2022-03-10T18:50:25.917 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:25.915945 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T18:50:25.917 INFO:tasks.rook.operator.smithi093.stdout:I0310 18:50:25.916109 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T18:50:26.469 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:26.471237 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi111 2022-03-10T18:50:26.470 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:26.471256 I | op-mon: mon c assigned to node smithi111 2022-03-10T18:50:26.475 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:26.476666 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-10T18:50:26.522 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:26.655 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:26.655769 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-10T18:50:26.655 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:26.656 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 112s 2022-03-10T18:50:27.089 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:27.089947 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-10T18:50:27.284 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:27.285627 I | op-mon: creating mon a 2022-03-10T18:50:27.284 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:27.285660 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:50:27.566 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:27.567483 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.161:6789"]}] data:a=172.21.15.161:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:-1] 2022-03-10T18:50:27.569 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:27.568091 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-10T18:50:27.570 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:27.568193 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-10T18:50:27.867 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:27.865625 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:50:27.868 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:27.866244 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:50:28.034 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:28.034874 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-10T18:50:28.351 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:28.351178 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-10T18:50:29.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:29.386188 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.161:6789"]}] data:a=172.21.15.161:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:0] 2022-03-10T18:50:29.385 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:29.386220 I | op-mon: waiting for mon quorum with [a] 2022-03-10T18:50:29.668 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:29.670533 I | op-mon: mon a is not yet running 2022-03-10T18:50:29.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:29.670562 I | op-mon: mons running: [] 2022-03-10T18:50:31.658 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:32.570 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:32.570 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 118s 2022-03-10T18:50:37.572 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:37.637 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:37.637 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m3s 2022-03-10T18:50:42.640 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:42.706 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:42.706 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m8s 2022-03-10T18:50:47.709 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:47.775 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:47.775 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m13s 2022-03-10T18:50:49.803 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:50:49.805945 I | op-mon: mons running: [a] 2022-03-10T18:50:52.778 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:52.869 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:52.869 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m18s 2022-03-10T18:50:57.872 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:50:57.945 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:50:57.945 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m23s 2022-03-10T18:51:02.948 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:03.253 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:03.254 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m29s 2022-03-10T18:51:08.256 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:08.511 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:08.512 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m34s 2022-03-10T18:51:10.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:10.262937 I | op-mon: mon a is not yet running 2022-03-10T18:51:10.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:10.262965 I | op-mon: mons running: [] 2022-03-10T18:51:11.096 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:11.099419 I | op-mon: Monitors in quorum: [a] 2022-03-10T18:51:11.097 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:11.099438 I | op-mon: mons created: 1 2022-03-10T18:51:11.303 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:11.305903 I | op-mon: waiting for mon quorum with [a] 2022-03-10T18:51:11.751 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:11.753525 I | op-mon: mons running: [a] 2022-03-10T18:51:11.971 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:11.963440 I | op-mon: Monitors in quorum: [a] 2022-03-10T18:51:11.972 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:11.963466 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:51:12.232 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:12.235087 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:51:12.232 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:12.235123 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:51:12.501 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:12.503890 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:51:12.501 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:12.503911 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:51:12.773 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:12.775604 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:51:12.773 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:12.775625 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:51:13.060 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.063210 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:51:13.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.063230 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T18:51:13.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.273961 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T18:51:13.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.273997 I | op-mon: creating mon b 2022-03-10T18:51:13.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.274008 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:51:13.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.274021 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:51:13.514 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.516724 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.161:6789","172.21.15.93:6789"]}] data:a=172.21.15.161:6789,b=172.21.15.93:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:0] 2022-03-10T18:51:13.514 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:13.780 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:13.781 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m39s 2022-03-10T18:51:13.864 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.866380 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:51:13.866 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.866576 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:51:13.919 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.921868 I | op-mon: 0 of 2 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-10T18:51:13.924 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.926726 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T18:51:13.935 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:13.938351 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T18:51:14.011 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:14.013933 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-10T18:51:15.435 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:15.437970 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.161:6789","172.21.15.93:6789"]}] data:a=172.21.15.161:6789,b=172.21.15.93:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:1] 2022-03-10T18:51:15.435 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:15.438003 I | op-mon: waiting for mon quorum with [a b] 2022-03-10T18:51:16.193 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:16.196124 I | op-mon: mon b is not yet running 2022-03-10T18:51:16.196 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:16.196582 I | op-mon: mons running: [a] 2022-03-10T18:51:16.429 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:16.432066 I | op-mon: Monitors in quorum: [a] 2022-03-10T18:51:16.429 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:16.432097 I | op-mon: mons created: 2 2022-03-10T18:51:16.647 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:16.650146 I | op-mon: waiting for mon quorum with [a b] 2022-03-10T18:51:17.132 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:17.135013 I | op-mon: mon b is not yet running 2022-03-10T18:51:17.132 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:17.135065 I | op-mon: mons running: [a] 2022-03-10T18:51:18.784 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:18.898 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:18.899 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m44s 2022-03-10T18:51:22.198 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:22.201519 I | op-mon: mon b is not yet running 2022-03-10T18:51:22.199 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:22.201548 I | op-mon: mons running: [a] 2022-03-10T18:51:23.902 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:23.968 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:23.968 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m49s 2022-03-10T18:51:27.210 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:27.213298 I | op-mon: mon b is not yet running 2022-03-10T18:51:27.210 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:27.213325 I | op-mon: mons running: [a] 2022-03-10T18:51:28.973 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:29.039 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:29.040 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 2m55s 2022-03-10T18:51:32.243 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:32.246495 I | op-mon: mon b is not yet running 2022-03-10T18:51:32.244 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:32.246524 I | op-mon: mons running: [a] 2022-03-10T18:51:34.041 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:34.115 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:34.115 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 3m 2022-03-10T18:51:37.468 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:37.471493 I | op-mon: mon b is not yet running 2022-03-10T18:51:37.612 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:37.471522 I | op-mon: mons running: [a] 2022-03-10T18:51:39.117 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:39.524 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:39.524 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 3m5s 2022-03-10T18:51:42.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:42.535599 I | op-mon: mon b is not yet running 2022-03-10T18:51:42.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:42.535626 I | op-mon: mons running: [a] 2022-03-10T18:51:44.526 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:44.587 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:44.588 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 0/1 ContainerCreating 0 3m10s 2022-03-10T18:51:48.234 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:48.238057 I | op-mon: mon b is not yet running 2022-03-10T18:51:48.235 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:48.238079 I | op-mon: mons running: [a] 2022-03-10T18:51:49.590 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T18:51:49.661 INFO:teuthology.orchestra.run.smithi093.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T18:51:49.661 INFO:teuthology.orchestra.run.smithi093.stdout:rook-ceph-tools-d6d7c985c-hbwqv 1/1 Running 0 3m15s 2022-03-10T18:51:49.662 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-10T18:51:49.662 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:51:50.188 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:51:50.189 INFO:teuthology.orchestra.run.smithi093.stderr:no valid command found; 10 closest matches: 2022-03-10T18:51:50.189 INFO:teuthology.orchestra.run.smithi093.stderr:pg stat 2022-03-10T18:51:50.189 INFO:teuthology.orchestra.run.smithi093.stderr:pg getmap 2022-03-10T18:51:50.190 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T18:51:50.190 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T18:51:50.190 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_pools_json 2022-03-10T18:51:50.190 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-pool [...] 2022-03-10T18:51:50.191 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-primary [] [...] 2022-03-10T18:51:50.191 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-osd [] [...] 2022-03-10T18:51:50.191 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls [] [...] 2022-03-10T18:51:50.191 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T18:51:50.192 INFO:teuthology.orchestra.run.smithi093.stderr:Error EINVAL: invalid command 2022-03-10T18:51:50.194 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 22 2022-03-10T18:51:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T18:51:53.248 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:53.251976 I | op-mon: mon b is not yet running 2022-03-10T18:51:53.249 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:53.252010 I | op-mon: mons running: [a] 2022-03-10T18:51:58.265 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:58.268807 I | op-mon: mons running: [a b] 2022-03-10T18:51:59.905 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:59.908897 I | op-mon: Monitors in quorum: [a b] 2022-03-10T18:51:59.906 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:51:59.908930 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:52:00.131 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:00.134656 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:52:00.131 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:00.134695 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:52:00.198 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:52:00.351 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:00.354662 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:52:00.351 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:00.354716 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:52:00.574 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:00.577832 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:52:00.574 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:00.577863 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:52:00.720 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:52:00.720 INFO:teuthology.orchestra.run.smithi093.stderr:no valid command found; 10 closest matches: 2022-03-10T18:52:00.720 INFO:teuthology.orchestra.run.smithi093.stderr:pg stat 2022-03-10T18:52:00.721 INFO:teuthology.orchestra.run.smithi093.stderr:pg getmap 2022-03-10T18:52:00.721 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T18:52:00.721 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T18:52:00.721 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_pools_json 2022-03-10T18:52:00.722 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-pool [...] 2022-03-10T18:52:00.722 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-primary [] [...] 2022-03-10T18:52:00.722 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-osd [] [...] 2022-03-10T18:52:00.722 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls [] [...] 2022-03-10T18:52:00.722 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T18:52:00.723 INFO:teuthology.orchestra.run.smithi093.stderr:Error EINVAL: invalid command 2022-03-10T18:52:00.726 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 22 2022-03-10T18:52:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T18:52:00.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:00.816003 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:52:00.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:00.816030 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T18:52:01.029 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.032051 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T18:52:01.029 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.032080 I | op-mon: creating mon c 2022-03-10T18:52:01.030 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.032088 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:52:01.030 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.032096 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:52:01.030 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.032102 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:52:01.050 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.053881 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.161:6789","172.21.15.93:6789","172.21.15.111:6789"]}] data:a=172.21.15.161:6789,b=172.21.15.93:6789,c=172.21.15.111:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:1] 2022-03-10T18:52:01.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.084682 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:52:01.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.084900 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:52:01.141 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.144847 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T18:52:01.155 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.158203 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T18:52:01.155 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.158237 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T18:52:01.236 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.240318 I | op-mon: mons running: [a b c] 2022-03-10T18:52:01.467 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.470802 I | op-mon: Monitors in quorum: [a b] 2022-03-10T18:52:01.473 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.476952 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T18:52:01.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.484191 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T18:52:01.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.484213 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T18:52:01.837 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:01.840938 I | op-mon: mons running: [a b c] 2022-03-10T18:52:02.060 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:02.063700 I | op-mon: Monitors in quorum: [a b] 2022-03-10T18:52:02.094 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:02.097775 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-10T18:52:02.835 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:02.838685 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.161:6789","172.21.15.93:6789","172.21.15.111:6789"]}] data:b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:2] 2022-03-10T18:52:02.836 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:02.838726 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T18:52:03.439 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:03.442619 I | op-mon: mons running: [a b c] 2022-03-10T18:52:03.663 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:03.667275 I | op-mon: Monitors in quorum: [a b] 2022-03-10T18:52:03.663 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:03.667304 I | op-mon: mons created: 3 2022-03-10T18:52:03.877 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:03.881163 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T18:52:04.036 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:04.039550 I | op-mon: mons running: [a b c] 2022-03-10T18:52:04.261 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:04.264885 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T18:52:04.262 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:04.264914 W | op-mon: monitor c is not in quorum list 2022-03-10T18:52:09.284 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:09.287925 I | op-mon: mons running: [a b c] 2022-03-10T18:52:09.501 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:09.505500 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T18:52:09.502 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:09.505533 W | op-mon: monitor c is not in quorum list 2022-03-10T18:52:10.730 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:52:11.246 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:52:11.248 INFO:teuthology.orchestra.run.smithi093.stderr:no valid command found; 10 closest matches: 2022-03-10T18:52:11.248 INFO:teuthology.orchestra.run.smithi093.stderr:pg stat 2022-03-10T18:52:11.248 INFO:teuthology.orchestra.run.smithi093.stderr:pg getmap 2022-03-10T18:52:11.249 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T18:52:11.249 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T18:52:11.249 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_pools_json 2022-03-10T18:52:11.249 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-pool [...] 2022-03-10T18:52:11.250 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-primary [] [...] 2022-03-10T18:52:11.250 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-osd [] [...] 2022-03-10T18:52:11.250 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls [] [...] 2022-03-10T18:52:11.250 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T18:52:11.251 INFO:teuthology.orchestra.run.smithi093.stderr:Error EINVAL: invalid command 2022-03-10T18:52:11.251 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 22 2022-03-10T18:52:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T18:52:14.522 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:14.526085 I | op-mon: mons running: [a b c] 2022-03-10T18:52:14.751 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:14.755980 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T18:52:14.752 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:14.755998 W | op-mon: monitor c is not in quorum list 2022-03-10T18:52:19.797 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:19.801361 I | op-mon: mons running: [a b c] 2022-03-10T18:52:20.035 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:20.039328 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T18:52:20.035 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:20.039371 W | op-mon: monitor c is not in quorum list 2022-03-10T18:52:21.254 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:52:21.729 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:52:21.729 INFO:teuthology.orchestra.run.smithi093.stderr:no valid command found; 10 closest matches: 2022-03-10T18:52:21.730 INFO:teuthology.orchestra.run.smithi093.stderr:pg stat 2022-03-10T18:52:21.730 INFO:teuthology.orchestra.run.smithi093.stderr:pg getmap 2022-03-10T18:52:21.731 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T18:52:21.731 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T18:52:21.731 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_pools_json 2022-03-10T18:52:21.732 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-pool [...] 2022-03-10T18:52:21.732 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-primary [] [...] 2022-03-10T18:52:21.732 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-osd [] [...] 2022-03-10T18:52:21.732 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls [] [...] 2022-03-10T18:52:21.732 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T18:52:21.733 INFO:teuthology.orchestra.run.smithi093.stderr:Error EINVAL: invalid command 2022-03-10T18:52:21.733 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 22 2022-03-10T18:52:21.735 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T18:52:25.058 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:25.061587 I | op-mon: mon c is not yet running 2022-03-10T18:52:25.058 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:25.061621 I | op-mon: mons running: [a b] 2022-03-10T18:52:30.080 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:30.084801 I | op-mon: mon c is not yet running 2022-03-10T18:52:30.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:30.084831 I | op-mon: mons running: [a b] 2022-03-10T18:52:31.737 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:52:32.274 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:52:32.274 INFO:teuthology.orchestra.run.smithi093.stderr:no valid command found; 10 closest matches: 2022-03-10T18:52:32.275 INFO:teuthology.orchestra.run.smithi093.stderr:pg stat 2022-03-10T18:52:32.275 INFO:teuthology.orchestra.run.smithi093.stderr:pg getmap 2022-03-10T18:52:32.275 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T18:52:32.275 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T18:52:32.276 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_pools_json 2022-03-10T18:52:32.276 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-pool [...] 2022-03-10T18:52:32.276 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-primary [] [...] 2022-03-10T18:52:32.276 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-osd [] [...] 2022-03-10T18:52:32.277 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls [] [...] 2022-03-10T18:52:32.277 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T18:52:32.277 INFO:teuthology.orchestra.run.smithi093.stderr:Error EINVAL: invalid command 2022-03-10T18:52:32.281 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 22 2022-03-10T18:52:32.283 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T18:52:35.102 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:35.106789 I | op-mon: mon c is not yet running 2022-03-10T18:52:35.103 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:35.106822 I | op-mon: mons running: [a b] 2022-03-10T18:52:40.125 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:40.129398 I | op-mon: mon c is not yet running 2022-03-10T18:52:40.125 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:40.129430 I | op-mon: mons running: [a b] 2022-03-10T18:52:42.285 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:52:42.785 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:52:42.785 INFO:teuthology.orchestra.run.smithi093.stderr:no valid command found; 10 closest matches: 2022-03-10T18:52:42.786 INFO:teuthology.orchestra.run.smithi093.stderr:pg stat 2022-03-10T18:52:42.786 INFO:teuthology.orchestra.run.smithi093.stderr:pg getmap 2022-03-10T18:52:42.786 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T18:52:42.786 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T18:52:42.787 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_pools_json 2022-03-10T18:52:42.787 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-pool [...] 2022-03-10T18:52:42.787 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-primary [] [...] 2022-03-10T18:52:42.787 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-osd [] [...] 2022-03-10T18:52:42.788 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls [] [...] 2022-03-10T18:52:42.788 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T18:52:42.788 INFO:teuthology.orchestra.run.smithi093.stderr:Error EINVAL: invalid command 2022-03-10T18:52:42.789 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 22 2022-03-10T18:52:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T18:52:45.147 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:45.152545 I | op-mon: mon c is not yet running 2022-03-10T18:52:45.148 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:45.152583 I | op-mon: mons running: [a b] 2022-03-10T18:52:50.210 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:50.215426 I | op-mon: mon c is not yet running 2022-03-10T18:52:50.211 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:50.215452 I | op-mon: mons running: [a b] 2022-03-10T18:52:52.791 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:52:53.274 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:52:53.274 INFO:teuthology.orchestra.run.smithi093.stderr:no valid command found; 10 closest matches: 2022-03-10T18:52:53.275 INFO:teuthology.orchestra.run.smithi093.stderr:pg stat 2022-03-10T18:52:53.275 INFO:teuthology.orchestra.run.smithi093.stderr:pg getmap 2022-03-10T18:52:53.275 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T18:52:53.275 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T18:52:53.276 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_pools_json 2022-03-10T18:52:53.276 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-pool [...] 2022-03-10T18:52:53.276 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-primary [] [...] 2022-03-10T18:52:53.276 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-osd [] [...] 2022-03-10T18:52:53.276 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls [] [...] 2022-03-10T18:52:53.277 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T18:52:53.277 INFO:teuthology.orchestra.run.smithi093.stderr:Error EINVAL: invalid command 2022-03-10T18:52:53.278 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 22 2022-03-10T18:52:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T18:52:55.267 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:55.271709 I | op-mon: mons running: [a b c] 2022-03-10T18:52:59.905 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:59.909858 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:52:59.906 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:52:59.909902 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:53:00.131 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:00.136730 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:53:00.132 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:00.136763 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:53:00.351 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:00.356790 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:53:00.352 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:00.356831 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:53:00.604 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:00.609503 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:53:00.605 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:00.609542 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:53:00.822 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:00.827454 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:53:00.823 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:00.827489 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T18:53:01.056 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:01.061220 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T18:53:01.056 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:01.061259 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T18:53:01.390 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:01.395516 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T18:53:01.723 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:01.728058 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T18:53:02.198 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:02.202763 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T18:53:02.624 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:02.629461 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T18:53:02.625 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:02.629487 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T18:53:02.983 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:02.988202 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T18:53:03.283 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:53:03.285 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:03.288772 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T18:53:03.286 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:03.288802 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T18:53:03.286 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:03.288810 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T18:53:03.635 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:03.640541 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T18:53:03.651 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:03.656295 I | op-mgr: start running mgr 2022-03-10T18:53:03.651 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:03.656331 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T18:53:03.748 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:53:03.748 INFO:teuthology.orchestra.run.smithi093.stderr:no valid command found; 10 closest matches: 2022-03-10T18:53:03.749 INFO:teuthology.orchestra.run.smithi093.stderr:pg stat 2022-03-10T18:53:03.749 INFO:teuthology.orchestra.run.smithi093.stderr:pg getmap 2022-03-10T18:53:03.749 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T18:53:03.750 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T18:53:03.750 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_pools_json 2022-03-10T18:53:03.750 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-pool [...] 2022-03-10T18:53:03.750 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-primary [] [...] 2022-03-10T18:53:03.751 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-osd [] [...] 2022-03-10T18:53:03.751 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls [] [...] 2022-03-10T18:53:03.751 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T18:53:03.751 INFO:teuthology.orchestra.run.smithi093.stderr:Error EINVAL: invalid command 2022-03-10T18:53:03.752 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 22 2022-03-10T18:53:03.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T18:53:04.383 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:04.388625 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi111": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:53:04.538 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:04.543066 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi111": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:53:04.570 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:04.575775 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi111": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:53:13.755 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:53:14.235 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:53:14.235 INFO:teuthology.orchestra.run.smithi093.stderr:no valid command found; 10 closest matches: 2022-03-10T18:53:14.236 INFO:teuthology.orchestra.run.smithi093.stderr:pg stat 2022-03-10T18:53:14.236 INFO:teuthology.orchestra.run.smithi093.stderr:pg getmap 2022-03-10T18:53:14.236 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T18:53:14.236 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T18:53:14.237 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_pools_json 2022-03-10T18:53:14.237 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-pool [...] 2022-03-10T18:53:14.237 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-primary [] [...] 2022-03-10T18:53:14.238 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls-by-osd [] [...] 2022-03-10T18:53:14.238 INFO:teuthology.orchestra.run.smithi093.stderr:pg ls [] [...] 2022-03-10T18:53:14.238 INFO:teuthology.orchestra.run.smithi093.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T18:53:14.238 INFO:teuthology.orchestra.run.smithi093.stderr:Error EINVAL: invalid command 2022-03-10T18:53:14.239 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 22 2022-03-10T18:53:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T18:53:24.242 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:53:24.772 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:53:24.773 INFO:teuthology.orchestra.run.smithi093.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-10T18:53:24.778 INFO:teuthology.orchestra.run.smithi093.stderr:command terminated with exit code 2 2022-03-10T18:53:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-10T18:53:25.414 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.419479 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-10T18:53:25.418 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.423517 I | op-mgr: setting services to point to mgr "a" 2022-03-10T18:53:25.473 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.473240 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T18:53:25.473 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.473296 I | op-mgr: successful modules: balancer 2022-03-10T18:53:25.537 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.542076 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T18:53:25.537 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.542100 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-10T18:53:25.538 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.542109 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T18:53:25.540 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.546341 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T18:53:25.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.549255 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-10T18:53:25.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.549282 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T18:53:25.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:25.549290 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T18:53:26.003 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:26.008517 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T18:53:26.003 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:26.008541 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T18:53:26.013 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:26.018700 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-10T18:53:26.014 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:26.018730 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-10T18:53:26.014 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:26.018743 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-10T18:53:26.468 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:26.473391 I | op-mgr: successful modules: dashboard 2022-03-10T18:53:26.471 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:26.476082 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T18:53:26.473 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:26.479301 I | op-mgr: successful modules: prometheus 2022-03-10T18:53:34.783 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status -f json 2022-03-10T18:53:34.876 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:34.882134 I | op-mgr: successful modules: orchestrator modules 2022-03-10T18:53:35.410 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:53:35.411 INFO:teuthology.orchestra.run.smithi093.stdout:{"available": true, "backend": "rook"} 2022-03-10T18:53:35.434 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-10T18:53:35.435 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-10T18:53:36.169 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch apply osd --all-available-devices 2022-03-10T18:53:36.847 INFO:teuthology.orchestra.run.smithi093.stdout:['Success'] 2022-03-10T18:53:36.850 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.853181 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-10T18:53:36.850 INFO:tasks.rook.operator.smithi093.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", AllowUnsupported: true}, 2022-03-10T18:53:36.850 INFO:tasks.rook.operator.smithi093.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-10T18:53:36.851 INFO:tasks.rook.operator.smithi093.stdout:   ... // 3 identical fields 2022-03-10T18:53:36.851 INFO:tasks.rook.operator.smithi093.stdout:   Config: nil, 2022-03-10T18:53:36.851 INFO:tasks.rook.operator.smithi093.stdout:   Selection: {}, 2022-03-10T18:53:36.851 INFO:tasks.rook.operator.smithi093.stdout:-  StorageClassDeviceSets: nil, 2022-03-10T18:53:36.852 INFO:tasks.rook.operator.smithi093.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T18:53:36.852 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.852 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi093-lv-1", 2022-03-10T18:53:36.852 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.853 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.853 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.853 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.853 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi093-lv-2", 2022-03-10T18:53:36.854 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.854 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.854 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.854 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.854 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi093-lv-3", 2022-03-10T18:53:36.855 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.855 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.855 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.856 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.856 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi093-lv-4", 2022-03-10T18:53:36.856 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.856 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.857 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.857 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.857 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi111-lv-1", 2022-03-10T18:53:36.857 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.861 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.861 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.861 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.862 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi111-lv-2", 2022-03-10T18:53:36.862 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.862 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.862 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.863 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.863 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi111-lv-3", 2022-03-10T18:53:36.863 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.863 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.863 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.864 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.864 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi111-lv-4", 2022-03-10T18:53:36.864 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.864 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.864 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.865 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.865 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi161-lv-1", 2022-03-10T18:53:36.865 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.865 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.866 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.866 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.866 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi161-lv-2", 2022-03-10T18:53:36.867 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.867 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.867 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.867 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.868 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi161-lv-3", 2022-03-10T18:53:36.868 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.868 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.868 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.868 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:53:36.869 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi161-lv-4", 2022-03-10T18:53:36.869 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:53:36.869 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T18:53:36.869 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.870 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:53:36.870 INFO:tasks.rook.operator.smithi093.stdout:   }, 2022-03-10T18:53:36.870 INFO:tasks.rook.operator.smithi093.stdout:   Annotations: nil, 2022-03-10T18:53:36.870 INFO:tasks.rook.operator.smithi093.stdout:   Labels: nil, 2022-03-10T18:53:36.870 INFO:tasks.rook.operator.smithi093.stdout:   ... // 20 identical fields 2022-03-10T18:53:36.871 INFO:tasks.rook.operator.smithi093.stdout:  } 2022-03-10T18:53:36.871 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.853275 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T18:53:36.871 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.853436 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-10T18:53:36.871 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.853457 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-10T18:53:36.872 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.853466 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-10T18:53:36.872 INFO:tasks.rook.operator.smithi093.stdout:I0310 18:53:36.853495 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T18:53:36.872 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.853662 I | operator: successfully started the controller-runtime manager 2022-03-10T18:53:36.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.875078 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T18:53:36.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.875112 I | operator: watching all namespaces for Ceph CRs 2022-03-10T18:53:36.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.875151 I | operator: setting up schemes 2022-03-10T18:53:36.874 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-10T18:53:36.875 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:53:36.875 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:36.878700 I | operator: setting up the controller-runtime manager 2022-03-10T18:53:37.176 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.181938 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T18:53:37.180 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184278 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T18:53:37.180 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184316 I | ceph-cluster-controller: successfully started 2022-03-10T18:53:37.181 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184387 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T18:53:37.181 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184413 I | ceph-crashcollector-controller: successfully started 2022-03-10T18:53:37.181 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184432 I | ceph-block-pool-controller: successfully started 2022-03-10T18:53:37.182 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184452 I | ceph-object-store-user-controller: successfully started 2022-03-10T18:53:37.182 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184467 I | ceph-object-realm-controller: successfully started 2022-03-10T18:53:37.183 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184487 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T18:53:37.183 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184499 I | ceph-object-zone-controller: successfully started 2022-03-10T18:53:37.183 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184636 I | ceph-object-controller: successfully started 2022-03-10T18:53:37.183 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184675 I | ceph-file-controller: successfully started 2022-03-10T18:53:37.184 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184702 I | ceph-nfs-controller: successfully started 2022-03-10T18:53:37.184 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184730 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T18:53:37.184 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184758 I | ceph-client-controller: successfully started 2022-03-10T18:53:37.184 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184771 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T18:53:37.184 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184824 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T18:53:37.185 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184848 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T18:53:37.185 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184860 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T18:53:37.185 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184882 I | ceph-bucket-topic: successfully started 2022-03-10T18:53:37.185 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184914 I | ceph-bucket-notification: successfully started 2022-03-10T18:53:37.186 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184933 I | ceph-bucket-notification: successfully started 2022-03-10T18:53:37.186 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.184942 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T18:53:37.186 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.185741 I | operator: starting the controller-runtime manager 2022-03-10T18:53:37.285 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.290947 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T18:53:37.286 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.291109 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T18:53:37.286 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.291180 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T18:53:37.286 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.291318 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T18:53:37.294 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.296143 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:53:37.294 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.296203 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T18:53:37.295 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.296215 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T18:53:37.295 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.296291 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T18:53:37.295 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.297712 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T18:53:37.295 INFO:tasks.rook.operator.smithi093.stdout:I0310 18:53:37.297768 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T18:53:37.296 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.299511 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:53:37.306 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.313006 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T18:53:37.313 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318627 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T18:53:37.314 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318646 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T18:53:37.314 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318651 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T18:53:37.314 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318657 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T18:53:37.314 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318663 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T18:53:37.315 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318668 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T18:53:37.315 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318674 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T18:53:37.315 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318681 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T18:53:37.315 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318687 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T18:53:37.316 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318693 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T18:53:37.316 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318701 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T18:53:37.316 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318708 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T18:53:37.316 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318717 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T18:53:37.317 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318721 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T18:53:37.317 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318728 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T18:53:37.317 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318735 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T18:53:37.318 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318793 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:53:37.318 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.318809 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:53:37.322 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:37.327614 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210... 2022-03-10T18:53:37.424 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:53:37.442 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T18:53:37.442 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T18:53:42.916 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:42.921922 I | ceph-spec: detected ceph image version: "17.0.0-11084 quincy" 2022-03-10T18:53:42.919 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:42.922790 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T18:53:42.919 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:42.922882 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11084 quincy", pursuing 2022-03-10T18:53:42.976 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:42.982396 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:53:43.107 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.113054 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:53:43.111 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.113408 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:53:43.825 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.831019 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11084 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210" 2022-03-10T18:53:43.956 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961124 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T18:53:43.956 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961160 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T18:53:43.956 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961170 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T18:53:43.957 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961174 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T18:53:43.957 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961179 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T18:53:43.957 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961183 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T18:53:43.957 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961188 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T18:53:43.958 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961193 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T18:53:43.958 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961198 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T18:53:43.958 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961203 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T18:53:43.958 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961208 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T18:53:43.959 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961213 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T18:53:43.959 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961218 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T18:53:43.959 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961223 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T18:53:43.960 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961227 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T18:53:43.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961233 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T18:53:43.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961238 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T18:53:43.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961243 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T18:53:43.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961248 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T18:53:43.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961253 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T18:53:43.962 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961261 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T18:53:43.962 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961266 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T18:53:43.962 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961270 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T18:53:43.962 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961276 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T18:53:43.963 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:43.961281 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T18:53:43.999 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.003218 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T18:53:44.005 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.011956 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:53:44.006 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.012146 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:53:44.009 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.012737 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:53:44.009 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.012750 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:53:44.009 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.012756 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:53:44.010 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.012761 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:53:44.010 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.012772 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T18:53:44.161 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.167805 I | op-mon: start running mons 2022-03-10T18:53:44.186 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.192588 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:53:44.187 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.192633 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:53:44.187 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.192648 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T18:53:44.190 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.193879 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:53:44.282 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.287834 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.111:6789","172.21.15.161:6789","172.21.15.93:6789"]}] data:b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:2] 2022-03-10T18:53:44.292 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.298416 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T18:53:44.750 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.755450 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:53:44.750 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:44.755814 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:53:45.466 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:45.472377 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:53:45.467 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:45.472412 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:53:45.467 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:45.472422 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T18:53:45.691 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:45.697785 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:53:45.692 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:45.697803 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:53:45.692 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:45.697807 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T18:53:45.769 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:45.775772 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T18:53:46.902 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:46.908713 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T18:53:46.908 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:46.914455 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T18:53:46.908 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:46.914478 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T18:53:46.912 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:46.919110 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T18:53:47.300 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:47.306730 I | op-mon: targeting the mon count 3 2022-03-10T18:53:47.305 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:47.312059 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:53:47.443 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:53:47.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:47.659053 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:53:47.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:47.659090 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:53:47.939 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:53:47.971 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T18:53:47.972 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T18:53:47.986 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:47.992770 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:53:47.986 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:47.992790 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:53:48.373 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:48.379490 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:53:48.374 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:48.379528 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:53:48.708 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:48.714575 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:53:48.709 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:48.714622 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T18:53:49.048 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.054002 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T18:53:49.048 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.054026 I | op-mon: checking for basic quorum with existing mons 2022-03-10T18:53:49.048 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.054034 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:53:49.049 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.054041 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:53:49.049 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.054045 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:53:49.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.067615 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.93:6789","172.21.15.111:6789","172.21.15.161:6789"]}] data:b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:2] 2022-03-10T18:53:49.103 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.105389 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:53:49.103 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.105647 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:53:49.115 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.121954 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T18:53:49.124 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.130212 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T18:53:49.124 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.130232 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:53:49.505 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.510976 I | op-mon: mons running: [b c a] 2022-03-10T18:53:49.920 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.926686 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:53:49.935 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.941699 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T18:53:49.951 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.957968 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T18:53:49.952 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:49.957989 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:53:50.118 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:50.124650 I | op-mon: mons running: [b c a] 2022-03-10T18:53:50.546 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:50.552583 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:53:50.552 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:50.558190 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T18:53:50.560 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:50.566778 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T18:53:50.560 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:50.566801 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:53:50.704 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:50.711095 I | op-mon: mons running: [b c a] 2022-03-10T18:53:51.106 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:51.112568 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:53:51.107 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:51.112598 I | op-mon: mons created: 3 2022-03-10T18:53:51.522 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:51.528669 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:53:51.540 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:51.547088 I | op-mon: mons running: [b c a] 2022-03-10T18:53:51.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:51.998394 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:53:51.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:51.998429 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T18:53:52.397 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:52.403491 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T18:53:52.820 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:52.826795 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T18:53:53.247 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:53.253387 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T18:53:53.698 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:53.704778 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T18:53:53.699 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:53.704808 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T18:53:54.134 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:54.140680 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T18:53:54.566 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:54.571940 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T18:53:54.566 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:54.571979 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T18:53:54.566 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:54.571992 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T18:53:54.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:54.998229 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T18:53:55.022 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.028416 I | op-mgr: start running mgr 2022-03-10T18:53:55.022 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.028458 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T18:53:55.479 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.485703 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T18:53:55.485 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.492037 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T18:53:55.488 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.494990 I | op-mgr: setting services to point to mgr "a" 2022-03-10T18:53:55.679 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.685558 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T18:53:55.774 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.685669 I | op-mgr: successful modules: balancer 2022-03-10T18:53:55.775 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.699544 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T18:53:55.776 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.699571 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T18:53:55.776 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.721811 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T18:53:55.877 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:55.883542 I | op-osd: creating 1 new PVCs for device set "smithi093-lv-1" 2022-03-10T18:53:56.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.087580 I | op-osd: successfully provisioned PVC "smithi093-lv-1-data-06k2m2" 2022-03-10T18:53:56.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.087607 I | op-osd: creating 1 new PVCs for device set "smithi093-lv-2" 2022-03-10T18:53:56.299 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.305711 I | op-osd: successfully provisioned PVC "smithi093-lv-2-data-0556qk" 2022-03-10T18:53:56.300 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.305771 I | op-osd: creating 1 new PVCs for device set "smithi093-lv-3" 2022-03-10T18:53:56.481 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.487115 I | op-osd: successfully provisioned PVC "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:53:56.481 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.487150 I | op-osd: creating 1 new PVCs for device set "smithi093-lv-4" 2022-03-10T18:53:56.683 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.688320 I | op-osd: successfully provisioned PVC "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:53:56.684 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.688448 I | op-osd: creating 1 new PVCs for device set "smithi111-lv-1" 2022-03-10T18:53:56.881 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.886919 I | op-osd: successfully provisioned PVC "smithi111-lv-1-data-0z2jls" 2022-03-10T18:53:56.881 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:56.886969 I | op-osd: creating 1 new PVCs for device set "smithi111-lv-2" 2022-03-10T18:53:57.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.087116 I | op-osd: successfully provisioned PVC "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:53:57.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.087174 I | op-osd: creating 1 new PVCs for device set "smithi111-lv-3" 2022-03-10T18:53:57.280 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.286212 I | op-osd: successfully provisioned PVC "smithi111-lv-3-data-02m2mz" 2022-03-10T18:53:57.280 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.286251 I | op-osd: creating 1 new PVCs for device set "smithi111-lv-4" 2022-03-10T18:53:57.381 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.388020 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T18:53:57.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.389193 I | op-mgr: successful modules: prometheus 2022-03-10T18:53:57.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.389405 I | op-mgr: successful modules: dashboard 2022-03-10T18:53:57.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.486972 I | op-osd: successfully provisioned PVC "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:53:57.481 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.487020 I | op-osd: creating 1 new PVCs for device set "smithi161-lv-1" 2022-03-10T18:53:57.684 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.690620 I | op-osd: successfully provisioned PVC "smithi161-lv-1-data-0vstrh" 2022-03-10T18:53:57.684 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.690658 I | op-osd: creating 1 new PVCs for device set "smithi161-lv-2" 2022-03-10T18:53:57.889 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.895713 I | op-osd: successfully provisioned PVC "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:53:57.890 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:57.895748 I | op-osd: creating 1 new PVCs for device set "smithi161-lv-3" 2022-03-10T18:53:57.974 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:53:58.080 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:58.086347 I | op-osd: successfully provisioned PVC "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:53:58.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:58.086382 I | op-osd: creating 1 new PVCs for device set "smithi161-lv-4" 2022-03-10T18:53:58.280 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:58.286549 I | op-osd: successfully provisioned PVC "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:53:58.311 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:58.317856 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-1-data-06k2m2" 2022-03-10T18:53:58.553 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:53:58.574 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T18:53:58.575 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T18:53:58.736 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:58.742678 I | op-osd: started OSD provisioning job for PVC "smithi093-lv-1-data-06k2m2" 2022-03-10T18:53:58.736 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:58.742704 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-2-data-0556qk" 2022-03-10T18:53:58.952 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:58.958984 I | op-mgr: successful modules: orchestrator modules 2022-03-10T18:53:59.114 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:59.121012 I | op-osd: started OSD provisioning job for PVC "smithi093-lv-2-data-0556qk" 2022-03-10T18:53:59.115 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:59.121041 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:53:59.499 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:59.505080 I | op-osd: started OSD provisioning job for PVC "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:53:59.499 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:53:59.505113 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:54:00.588 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:00.594754 I | op-osd: started OSD provisioning job for PVC "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:54:00.589 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:00.594785 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-1-data-0z2jls" 2022-03-10T18:54:01.621 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:01.627155 I | op-osd: started OSD provisioning job for PVC "smithi111-lv-1-data-0z2jls" 2022-03-10T18:54:01.621 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:01.627202 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:54:02.096 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:02.102229 I | op-osd: started OSD provisioning job for PVC "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:54:02.096 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:02.102252 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-3-data-02m2mz" 2022-03-10T18:54:03.888 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:03.892588 I | op-osd: started OSD provisioning job for PVC "smithi111-lv-3-data-02m2mz" 2022-03-10T18:54:03.889 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:03.892618 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:54:05.098 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:05.103385 I | op-osd: started OSD provisioning job for PVC "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:54:05.101 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:05.103551 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-1-data-0vstrh" 2022-03-10T18:54:05.841 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:05.844765 I | op-osd: started OSD provisioning job for PVC "smithi161-lv-1-data-0vstrh" 2022-03-10T18:54:05.841 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:05.844796 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:54:06.675 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:06.678738 I | op-osd: started OSD provisioning job for PVC "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:54:06.675 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:06.678759 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:54:08.313 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:08.317478 I | op-osd: started OSD provisioning job for PVC "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:54:08.313 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:08.317513 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:54:08.576 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:54:09.745 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.747517 I | op-osd: started OSD provisioning job for PVC "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:54:09.745 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.747558 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T18:54:09.746 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.747565 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T18:54:09.901 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:54:09.918 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T18:54:09.919 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T18:54:09.987 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990315 I | op-osd: OSD orchestration status for PVC smithi093-lv-1-data-06k2m2 is "starting" 2022-03-10T18:54:09.990 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990351 I | op-osd: OSD orchestration status for PVC smithi093-lv-2-data-0556qk is "starting" 2022-03-10T18:54:09.990 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990365 I | op-osd: OSD orchestration status for PVC smithi093-lv-3-data-0q2rqj is "starting" 2022-03-10T18:54:09.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990372 I | op-osd: OSD orchestration status for PVC smithi093-lv-4-data-0hwwsd is "starting" 2022-03-10T18:54:09.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990380 I | op-osd: OSD orchestration status for PVC smithi111-lv-1-data-0z2jls is "starting" 2022-03-10T18:54:09.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990388 I | op-osd: OSD orchestration status for PVC smithi111-lv-2-data-0rqjrb is "starting" 2022-03-10T18:54:09.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990396 I | op-osd: OSD orchestration status for PVC smithi111-lv-3-data-02m2mz is "starting" 2022-03-10T18:54:09.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990404 I | op-osd: OSD orchestration status for PVC smithi111-lv-4-data-0cfp2j is "starting" 2022-03-10T18:54:09.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990412 I | op-osd: OSD orchestration status for PVC smithi161-lv-1-data-0vstrh is "starting" 2022-03-10T18:54:09.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990419 I | op-osd: OSD orchestration status for PVC smithi161-lv-2-data-0b6h5v is "starting" 2022-03-10T18:54:09.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990427 I | op-osd: OSD orchestration status for PVC smithi161-lv-3-data-0dw8c4 is "starting" 2022-03-10T18:54:09.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:09.990435 I | op-osd: OSD orchestration status for PVC smithi161-lv-4-data-0m9dpg is "starting" 2022-03-10T18:54:16.031 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:16.034510 I | op-osd: OSD orchestration status for node smithi093-lv-1-data-06k2m2 is "orchestrating" 2022-03-10T18:54:16.361 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:16.107521 I | op-osd: OSD orchestration status for node smithi093-lv-2-data-0556qk is "orchestrating" 2022-03-10T18:54:16.461 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:16.464057 I | op-osd: OSD orchestration status for PVC smithi093-lv-1-data-06k2m2 is "orchestrating" 2022-03-10T18:54:16.716 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:16.719174 I | op-osd: OSD orchestration status for node smithi111-lv-2-data-0rqjrb is "orchestrating" 2022-03-10T18:54:16.804 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:16.807002 I | op-osd: OSD orchestration status for node smithi111-lv-1-data-0z2jls is "orchestrating" 2022-03-10T18:54:16.851 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:16.854700 I | op-osd: OSD orchestration status for PVC smithi093-lv-2-data-0556qk is "orchestrating" 2022-03-10T18:54:17.164 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:17.167627 I | op-osd: OSD orchestration status for PVC smithi111-lv-2-data-0rqjrb is "orchestrating" 2022-03-10T18:54:17.528 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:17.531593 I | op-osd: OSD orchestration status for PVC smithi111-lv-1-data-0z2jls is "orchestrating" 2022-03-10T18:54:18.708 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:18.710675 I | op-osd: OSD orchestration status for node smithi111-lv-4-data-0cfp2j is "orchestrating" 2022-03-10T18:54:18.947 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:18.950317 I | op-osd: OSD orchestration status for node smithi093-lv-3-data-0q2rqj is "orchestrating" 2022-03-10T18:54:19.091 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:19.094141 I | op-osd: OSD orchestration status for PVC smithi111-lv-4-data-0cfp2j is "orchestrating" 2022-03-10T18:54:19.248 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:19.251236 I | op-osd: OSD orchestration status for node smithi093-lv-4-data-0hwwsd is "orchestrating" 2022-03-10T18:54:19.397 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:19.400224 I | op-osd: OSD orchestration status for PVC smithi093-lv-3-data-0q2rqj is "orchestrating" 2022-03-10T18:54:19.630 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:19.632877 I | op-osd: OSD orchestration status for PVC smithi093-lv-4-data-0hwwsd is "orchestrating" 2022-03-10T18:54:19.920 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:54:20.224 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:20.226322 I | op-osd: OSD orchestration status for node smithi111-lv-3-data-02m2mz is "orchestrating" 2022-03-10T18:54:20.656 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:20.658556 I | op-osd: OSD orchestration status for PVC smithi111-lv-3-data-02m2mz is "orchestrating" 2022-03-10T18:54:20.790 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:54:20.816 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":5,"num_osds":3,"num_up_osds":0,"osd_up_since":0,"num_in_osds":3,"osd_in_since":1646938460,"num_remapped_pgs":0} 2022-03-10T18:54:20.816 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T18:54:24.142 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:24.143962 I | op-osd: OSD orchestration status for PVC smithi093-lv-1-data-06k2m2 is "completed" 2022-03-10T18:54:24.142 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:24.143994 I | op-osd: creating OSD 0 on PVC "smithi093-lv-1-data-06k2m2" 2022-03-10T18:54:24.143 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:24.144005 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-1-data-06k2m2" 2022-03-10T18:54:25.164 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:25.165103 I | op-osd: OSD orchestration status for PVC smithi093-lv-2-data-0556qk is "completed" 2022-03-10T18:54:25.165 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:25.165286 I | op-osd: creating OSD 1 on PVC "smithi093-lv-2-data-0556qk" 2022-03-10T18:54:25.165 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:25.165340 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-2-data-0556qk" 2022-03-10T18:54:26.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:26.385930 I | op-osd: OSD orchestration status for node smithi161-lv-1-data-0vstrh is "orchestrating" 2022-03-10T18:54:26.386 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:26.386281 I | op-osd: OSD orchestration status for PVC smithi111-lv-2-data-0rqjrb is "completed" 2022-03-10T18:54:26.386 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:26.386297 I | op-osd: creating OSD 2 on PVC "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:54:26.387 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:26.386308 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:54:28.601 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:28.602179 I | op-osd: OSD orchestration status for PVC smithi111-lv-1-data-0z2jls is "completed" 2022-03-10T18:54:28.601 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:28.602308 I | op-osd: creating OSD 3 on PVC "smithi111-lv-1-data-0z2jls" 2022-03-10T18:54:28.602 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:28.602465 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-1-data-0z2jls" 2022-03-10T18:54:29.755 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:29.755794 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi111": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:54:30.114 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:30.114280 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi111": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:54:30.818 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:54:31.070 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:31.067826 I | op-osd: OSD orchestration status for PVC smithi161-lv-1-data-0vstrh is "orchestrating" 2022-03-10T18:54:31.077 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:31.078280 I | op-osd: OSD orchestration status for node smithi161-lv-3-data-0dw8c4 is "orchestrating" 2022-03-10T18:54:31.077 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:31.078321 I | op-osd: OSD orchestration status for node smithi161-lv-2-data-0b6h5v is "orchestrating" 2022-03-10T18:54:31.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:31.078892 I | op-osd: OSD orchestration status for PVC smithi161-lv-2-data-0b6h5v is "orchestrating" 2022-03-10T18:54:31.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:31.078910 I | op-osd: OSD orchestration status for PVC smithi161-lv-3-data-0dw8c4 is "orchestrating" 2022-03-10T18:54:31.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:31.079093 I | op-osd: OSD orchestration status for PVC smithi111-lv-4-data-0cfp2j is "completed" 2022-03-10T18:54:31.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:31.079105 I | op-osd: creating OSD 4 on PVC "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:54:31.082 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:31.079113 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:54:32.289 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:32.290651 I | op-osd: OSD orchestration status for node smithi161-lv-4-data-0m9dpg is "orchestrating" 2022-03-10T18:54:32.294 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:32.292476 I | op-osd: OSD orchestration status for PVC smithi093-lv-4-data-0hwwsd is "completed" 2022-03-10T18:54:32.294 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:32.292510 I | op-osd: creating OSD 5 on PVC "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:54:32.294 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:32.292617 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:54:33.105 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:54:33.129 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":12,"num_osds":10,"num_up_osds":0,"osd_up_since":0,"num_in_osds":10,"osd_in_since":1646938469,"num_remapped_pgs":0} 2022-03-10T18:54:33.129 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T18:54:33.796 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:33.795329 I | op-osd: OSD orchestration status for PVC smithi111-lv-3-data-02m2mz is "completed" 2022-03-10T18:54:33.796 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:33.795399 I | op-osd: creating OSD 7 on PVC "smithi111-lv-3-data-02m2mz" 2022-03-10T18:54:33.796 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:33.795428 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-3-data-02m2mz" 2022-03-10T18:54:35.380 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:35.380226 I | op-osd: OSD orchestration status for PVC smithi161-lv-4-data-0m9dpg is "orchestrating" 2022-03-10T18:54:35.386 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:35.380631 I | op-osd: OSD orchestration status for PVC smithi093-lv-3-data-0q2rqj is "completed" 2022-03-10T18:54:35.387 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:35.381101 I | op-osd: creating OSD 6 on PVC "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:54:35.387 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:35.381123 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:54:37.838 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:37.838832 I | op-osd: OSD orchestration status for PVC smithi161-lv-2-data-0b6h5v is "completed" 2022-03-10T18:54:37.839 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:37.838866 I | op-osd: creating OSD 8 on PVC "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:54:37.839 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:37.838889 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:54:38.918 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:38.917366 I | op-osd: OSD orchestration status for PVC smithi161-lv-1-data-0vstrh is "completed" 2022-03-10T18:54:38.918 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:38.917392 I | op-osd: creating OSD 9 on PVC "smithi161-lv-1-data-0vstrh" 2022-03-10T18:54:38.919 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:38.917406 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-1-data-0vstrh" 2022-03-10T18:54:40.567 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:40.567141 I | op-osd: OSD orchestration status for PVC smithi161-lv-3-data-0dw8c4 is "completed" 2022-03-10T18:54:40.568 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:40.567172 I | op-osd: creating OSD 10 on PVC "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:54:40.568 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:40.567187 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:54:42.182 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:42.182270 I | op-osd: OSD orchestration status for PVC smithi161-lv-4-data-0m9dpg is "completed" 2022-03-10T18:54:42.182 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:42.182299 I | op-osd: creating OSD 11 on PVC "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:54:42.183 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:42.182319 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:54:43.131 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:54:45.200 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:45.200397 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T18:54:45.201 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:54:45.200424 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T18:54:45.363 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:54:45.383 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646938473,"num_remapped_pgs":0} 2022-03-10T18:54:45.384 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T18:54:55.385 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:54:56.185 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:54:56.204 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646938473,"num_remapped_pgs":0} 2022-03-10T18:54:56.204 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T18:55:06.205 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:55:06.764 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:06.783 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":22,"num_osds":12,"num_up_osds":5,"osd_up_since":1646938505,"num_in_osds":12,"osd_in_since":1646938473,"num_remapped_pgs":0} 2022-03-10T18:55:06.784 INFO:tasks.rook: have 5/12 OSDs 2022-03-10T18:55:16.785 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd stat -f json 2022-03-10T18:55:17.419 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:17.439 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":28,"num_osds":12,"num_up_osds":12,"osd_up_since":1646938511,"num_in_osds":12,"osd_in_since":1646938473,"num_remapped_pgs":0} 2022-03-10T18:55:17.440 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-10T18:55:17.440 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- cat /etc/ceph/ceph.conf 2022-03-10T18:55:17.625 INFO:teuthology.orchestra.run.smithi093.stdout:[global] 2022-03-10T18:55:17.626 INFO:teuthology.orchestra.run.smithi093.stdout:mon_host = 172.21.15.93:6789,172.21.15.111:6789,172.21.15.161:6789 2022-03-10T18:55:17.626 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:17.626 INFO:teuthology.orchestra.run.smithi093.stdout:[client.admin] 2022-03-10T18:55:17.626 INFO:teuthology.orchestra.run.smithi093.stdout:keyring = /etc/ceph/keyring 2022-03-10T18:55:17.629 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- cat /etc/ceph/keyring 2022-03-10T18:55:17.826 INFO:teuthology.orchestra.run.smithi093.stdout:[client.admin] 2022-03-10T18:55:17.826 INFO:teuthology.orchestra.run.smithi093.stdout:key = AQBuSCpi7y4CCRAAANbkBxoHJ4nqaMQCGLyYLw== 2022-03-10T18:55:17.830 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /etc/ceph 2022-03-10T18:55:17.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2022-03-10T18:55:17.861 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph 2022-03-10T18:55:17.877 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:55:17.878 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T18:55:17.894 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:55:17.894 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/ceph/keyring 2022-03-10T18:55:17.946 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:55:17.946 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T18:55:17.962 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:55:17.963 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/keyring 2022-03-10T18:55:17.978 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:55:17.978 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T18:55:17.994 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:55:17.994 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/keyring 2022-03-10T18:55:18.052 INFO:tasks.rook:Setting up client nodes... 2022-03-10T18:55:18.053 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T18:55:18.796 INFO:teuthology.orchestra.run.smithi093.stdout:[client.a] 2022-03-10T18:55:18.797 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCWSSpixJHMJxAAskZrxqy7d++N4FjOu61TAQ== 2022-03-10T18:55:18.824 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2022-03-10T18:55:18.825 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-10T18:55:18.825 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-10T18:55:18.848 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T18:55:19.564 INFO:teuthology.orchestra.run.smithi093.stdout:[client.b] 2022-03-10T18:55:19.564 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCXSSpiYaZ9GhAAmbYylXeiWGMgvwsDHEvToQ== 2022-03-10T18:55:19.589 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-03-10T18:55:19.589 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-10T18:55:19.589 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-10T18:55:19.610 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T18:55:20.445 INFO:teuthology.orchestra.run.smithi093.stdout:[client.c] 2022-03-10T18:55:20.446 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCYSSpiZHnPDBAAMWE9cl/c8UitB3LDh4pL8w== 2022-03-10T18:55:20.471 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-03-10T18:55:20.471 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-10T18:55:20.472 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-10T18:55:20.497 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd dump --format=json 2022-03-10T18:55:21.008 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:21.008 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":28,"fsid":"2939e372-9a22-4d12-9ac6-4ccfb538e9cd","created":"2022-03-10T18:51:10.468251+0000","modified":"2022-03-10T18:55:13.567132+0000","last_up_change":"2022-03-10T18:55:11.106026+0000","last_in_change":"2022-03-10T18:54:33.759325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T18:55:05.021899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9d22befc-5088-4db6-ba40-146e86d78071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6809","nonce":2029997637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6810","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6811","nonce":2029997637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6815","nonce":2029997637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6813","nonce":2029997637}]},"public_addr":"172.21.15.93:6809/2029997637","cluster_addr":"172.21.15.93:6811/2029997637","heartbeat_back_addr":"172.21.15.93:6815/2029997637","heartbeat_front_addr":"172.21.15.93:6813/2029997637","state":["exists","up"]},{"osd":1,"uuid":"a1899423-2db6-45d5-b76a-1a336454c271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6801","nonce":1882492699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6803","nonce":1882492699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6807","nonce":1882492699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6805","nonce":1882492699}]},"public_addr":"172.21.15.93:6801/1882492699","cluster_addr":"172.21.15.93:6803/1882492699","heartbeat_back_addr":"172.21.15.93:6807/1882492699","heartbeat_front_addr":"172.21.15.93:6805/1882492699","state":["exists","up"]},{"osd":2,"uuid":"964955b1-bfd5-4f0d-94ac-def1377faf36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6803","nonce":4097433165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6805","nonce":4097433165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6809","nonce":4097433165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6807","nonce":4097433165}]},"public_addr":"172.21.15.111:6803/4097433165","cluster_addr":"172.21.15.111:6805/4097433165","heartbeat_back_addr":"172.21.15.111:6809/4097433165","heartbeat_front_addr":"172.21.15.111:6807/4097433165","state":["exists","up"]},{"osd":3,"uuid":"28521500-b718-4989-a00d-698f97d59192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6811","nonce":1593212704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6813","nonce":1593212704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6817","nonce":1593212704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6815","nonce":1593212704}]},"public_addr":"172.21.15.111:6811/1593212704","cluster_addr":"172.21.15.111:6813/1593212704","heartbeat_back_addr":"172.21.15.111:6817/1593212704","heartbeat_front_addr":"172.21.15.111:6815/1593212704","state":["exists","up"]},{"osd":4,"uuid":"800461e8-bcf8-4517-8730-b0e34360dbc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6819","nonce":1103376889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6821","nonce":1103376889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6825","nonce":1103376889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6823","nonce":1103376889}]},"public_addr":"172.21.15.111:6819/1103376889","cluster_addr":"172.21.15.111:6821/1103376889","heartbeat_back_addr":"172.21.15.111:6825/1103376889","heartbeat_front_addr":"172.21.15.111:6823/1103376889","state":["exists","up"]},{"osd":5,"uuid":"05f6c5ba-6061-4252-95ff-373cc1f963cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6816","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6817","nonce":2104620729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6819","nonce":2104620729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6823","nonce":2104620729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6820","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6821","nonce":2104620729}]},"public_addr":"172.21.15.93:6817/2104620729","cluster_addr":"172.21.15.93:6819/2104620729","heartbeat_back_addr":"172.21.15.93:6823/2104620729","heartbeat_front_addr":"172.21.15.93:6821/2104620729","state":["exists","up"]},{"osd":6,"uuid":"c0df1a7c-6359-49c5-b10f-8495554ecfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6825","nonce":692066384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6827","nonce":692066384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6831","nonce":692066384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6829","nonce":692066384}]},"public_addr":"172.21.15.93:6825/692066384","cluster_addr":"172.21.15.93:6827/692066384","heartbeat_back_addr":"172.21.15.93:6831/692066384","heartbeat_front_addr":"172.21.15.93:6829/692066384","state":["exists","up"]},{"osd":7,"uuid":"0a012d6b-a401-48f0-9562-4ce7a1df5190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6827","nonce":2069656674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6829","nonce":2069656674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6833","nonce":2069656674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6831","nonce":2069656674}]},"public_addr":"172.21.15.111:6827/2069656674","cluster_addr":"172.21.15.111:6829/2069656674","heartbeat_back_addr":"172.21.15.111:6833/2069656674","heartbeat_front_addr":"172.21.15.111:6831/2069656674","state":["exists","up"]},{"osd":8,"uuid":"508ca0d0-afad-4af4-9596-5e70f23cb384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6817","nonce":3645158221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6819","nonce":3645158221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6823","nonce":3645158221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6821","nonce":3645158221}]},"public_addr":"172.21.15.161:6817/3645158221","cluster_addr":"172.21.15.161:6819/3645158221","heartbeat_back_addr":"172.21.15.161:6823/3645158221","heartbeat_front_addr":"172.21.15.161:6821/3645158221","state":["exists","up"]},{"osd":9,"uuid":"02df1e47-4865-428c-8ecb-da66e1cc3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6801","nonce":3134211368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6803","nonce":3134211368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6807","nonce":3134211368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6805","nonce":3134211368}]},"public_addr":"172.21.15.161:6801/3134211368","cluster_addr":"172.21.15.161:6803/3134211368","heartbeat_back_addr":"172.21.15.161:6807/3134211368","heartbeat_front_addr":"172.21.15.161:6805/3134211368","state":["exists","up"]},{"osd":10,"uuid":"72166e4b-3d85-4fb5-85f2-111497256cf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6825","nonce":3615306489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6827","nonce":3615306489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6831","nonce":3615306489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6829","nonce":3615306489}]},"public_addr":"172.21.15.161:6825/3615306489","cluster_addr":"172.21.15.161:6827/3615306489","heartbeat_back_addr":"172.21.15.161:6831/3615306489","heartbeat_front_addr":"172.21.15.161:6829/3615306489","state":["exists","up"]},{"osd":11,"uuid":"d28fc35b-19c1-4133-b927-00955809ae38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6809","nonce":2588715091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6811","nonce":2588715091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6815","nonce":2588715091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6813","nonce":2588715091}]},"public_addr":"172.21.15.161:6809/2588715091","cluster_addr":"172.21.15.161:6811/2588715091","heartbeat_back_addr":"172.21.15.161:6815/2588715091","heartbeat_front_addr":"172.21.15.161:6813/2588715091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:54:58.907269+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:54:58.695544+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:54:57.412570+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:01.179071+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:02.393395+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:05.840639+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:09.422751+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:04.731462+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:07.963681+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:06.165232+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:09.455954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:07.881867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:0/1413700147":"2022-03-11T18:53:33.062389+0000","172.21.15.111:6800/2517481259":"2022-03-11T18:53:33.062389+0000","172.21.15.111:0/3279937313":"2022-03-11T18:53:33.062389+0000","172.21.15.111:6801/2517481259":"2022-03-11T18:53:33.062389+0000","172.21.15.111:0/3404334102":"2022-03-11T18:53:33.062389+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T18:55:21.026 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-10T18:55:05.021899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-10T18:55:21.026 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd pool get .mgr pg_num 2022-03-10T18:55:21.553 INFO:teuthology.orchestra.run.smithi093.stdout:pg_num: 1 2022-03-10T18:55:21.572 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-10T18:55:21.573 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-10T18:55:21.573 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph mgr dump --format=json 2022-03-10T18:55:22.168 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:22.185 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":12,"active_gid":24173,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":297140923},{"type":"v1","addr":"172.21.15.111:6801","nonce":297140923}]},"active_addr":"172.21.15.111:6801/297140923","active_change":"2022-03-10T18:53:33.096707+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://172.21.15.111:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":2992485791}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":945902020}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":915553589}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":2234958027}]}]}} 2022-03-10T18:55:22.188 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-10T18:55:22.188 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-10T18:55:22.189 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd dump --format=json 2022-03-10T18:55:22.716 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:22.717 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":28,"fsid":"2939e372-9a22-4d12-9ac6-4ccfb538e9cd","created":"2022-03-10T18:51:10.468251+0000","modified":"2022-03-10T18:55:13.567132+0000","last_up_change":"2022-03-10T18:55:11.106026+0000","last_in_change":"2022-03-10T18:54:33.759325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T18:55:05.021899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9d22befc-5088-4db6-ba40-146e86d78071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6809","nonce":2029997637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6810","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6811","nonce":2029997637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6815","nonce":2029997637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6813","nonce":2029997637}]},"public_addr":"172.21.15.93:6809/2029997637","cluster_addr":"172.21.15.93:6811/2029997637","heartbeat_back_addr":"172.21.15.93:6815/2029997637","heartbeat_front_addr":"172.21.15.93:6813/2029997637","state":["exists","up"]},{"osd":1,"uuid":"a1899423-2db6-45d5-b76a-1a336454c271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6801","nonce":1882492699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6803","nonce":1882492699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6807","nonce":1882492699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6805","nonce":1882492699}]},"public_addr":"172.21.15.93:6801/1882492699","cluster_addr":"172.21.15.93:6803/1882492699","heartbeat_back_addr":"172.21.15.93:6807/1882492699","heartbeat_front_addr":"172.21.15.93:6805/1882492699","state":["exists","up"]},{"osd":2,"uuid":"964955b1-bfd5-4f0d-94ac-def1377faf36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6803","nonce":4097433165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6805","nonce":4097433165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6809","nonce":4097433165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6807","nonce":4097433165}]},"public_addr":"172.21.15.111:6803/4097433165","cluster_addr":"172.21.15.111:6805/4097433165","heartbeat_back_addr":"172.21.15.111:6809/4097433165","heartbeat_front_addr":"172.21.15.111:6807/4097433165","state":["exists","up"]},{"osd":3,"uuid":"28521500-b718-4989-a00d-698f97d59192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6811","nonce":1593212704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6813","nonce":1593212704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6817","nonce":1593212704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6815","nonce":1593212704}]},"public_addr":"172.21.15.111:6811/1593212704","cluster_addr":"172.21.15.111:6813/1593212704","heartbeat_back_addr":"172.21.15.111:6817/1593212704","heartbeat_front_addr":"172.21.15.111:6815/1593212704","state":["exists","up"]},{"osd":4,"uuid":"800461e8-bcf8-4517-8730-b0e34360dbc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6819","nonce":1103376889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6821","nonce":1103376889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6825","nonce":1103376889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6823","nonce":1103376889}]},"public_addr":"172.21.15.111:6819/1103376889","cluster_addr":"172.21.15.111:6821/1103376889","heartbeat_back_addr":"172.21.15.111:6825/1103376889","heartbeat_front_addr":"172.21.15.111:6823/1103376889","state":["exists","up"]},{"osd":5,"uuid":"05f6c5ba-6061-4252-95ff-373cc1f963cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6816","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6817","nonce":2104620729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6819","nonce":2104620729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6823","nonce":2104620729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6820","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6821","nonce":2104620729}]},"public_addr":"172.21.15.93:6817/2104620729","cluster_addr":"172.21.15.93:6819/2104620729","heartbeat_back_addr":"172.21.15.93:6823/2104620729","heartbeat_front_addr":"172.21.15.93:6821/2104620729","state":["exists","up"]},{"osd":6,"uuid":"c0df1a7c-6359-49c5-b10f-8495554ecfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6825","nonce":692066384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6827","nonce":692066384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6831","nonce":692066384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6829","nonce":692066384}]},"public_addr":"172.21.15.93:6825/692066384","cluster_addr":"172.21.15.93:6827/692066384","heartbeat_back_addr":"172.21.15.93:6831/692066384","heartbeat_front_addr":"172.21.15.93:6829/692066384","state":["exists","up"]},{"osd":7,"uuid":"0a012d6b-a401-48f0-9562-4ce7a1df5190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6827","nonce":2069656674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6829","nonce":2069656674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6833","nonce":2069656674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6831","nonce":2069656674}]},"public_addr":"172.21.15.111:6827/2069656674","cluster_addr":"172.21.15.111:6829/2069656674","heartbeat_back_addr":"172.21.15.111:6833/2069656674","heartbeat_front_addr":"172.21.15.111:6831/2069656674","state":["exists","up"]},{"osd":8,"uuid":"508ca0d0-afad-4af4-9596-5e70f23cb384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6817","nonce":3645158221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6819","nonce":3645158221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6823","nonce":3645158221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6821","nonce":3645158221}]},"public_addr":"172.21.15.161:6817/3645158221","cluster_addr":"172.21.15.161:6819/3645158221","heartbeat_back_addr":"172.21.15.161:6823/3645158221","heartbeat_front_addr":"172.21.15.161:6821/3645158221","state":["exists","up"]},{"osd":9,"uuid":"02df1e47-4865-428c-8ecb-da66e1cc3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6801","nonce":3134211368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6803","nonce":3134211368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6807","nonce":3134211368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6805","nonce":3134211368}]},"public_addr":"172.21.15.161:6801/3134211368","cluster_addr":"172.21.15.161:6803/3134211368","heartbeat_back_addr":"172.21.15.161:6807/3134211368","heartbeat_front_addr":"172.21.15.161:6805/3134211368","state":["exists","up"]},{"osd":10,"uuid":"72166e4b-3d85-4fb5-85f2-111497256cf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6825","nonce":3615306489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6827","nonce":3615306489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6831","nonce":3615306489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6829","nonce":3615306489}]},"public_addr":"172.21.15.161:6825/3615306489","cluster_addr":"172.21.15.161:6827/3615306489","heartbeat_back_addr":"172.21.15.161:6831/3615306489","heartbeat_front_addr":"172.21.15.161:6829/3615306489","state":["exists","up"]},{"osd":11,"uuid":"d28fc35b-19c1-4133-b927-00955809ae38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6809","nonce":2588715091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6811","nonce":2588715091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6815","nonce":2588715091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6813","nonce":2588715091}]},"public_addr":"172.21.15.161:6809/2588715091","cluster_addr":"172.21.15.161:6811/2588715091","heartbeat_back_addr":"172.21.15.161:6815/2588715091","heartbeat_front_addr":"172.21.15.161:6813/2588715091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:54:58.907269+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:54:58.695544+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:54:57.412570+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:01.179071+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:02.393395+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:05.840639+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:09.422751+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:04.731462+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:07.963681+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:06.165232+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:09.455954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:07.881867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:0/1413700147":"2022-03-11T18:53:33.062389+0000","172.21.15.111:6800/2517481259":"2022-03-11T18:53:33.062389+0000","172.21.15.111:0/3279937313":"2022-03-11T18:53:33.062389+0000","172.21.15.111:6801/2517481259":"2022-03-11T18:53:33.062389+0000","172.21.15.111:0/3404334102":"2022-03-11T18:53:33.062389+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T18:55:22.735 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-10T18:55:22.736 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd dump --format=json 2022-03-10T18:55:23.254 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:23.254 INFO:teuthology.orchestra.run.smithi093.stdout:{"epoch":28,"fsid":"2939e372-9a22-4d12-9ac6-4ccfb538e9cd","created":"2022-03-10T18:51:10.468251+0000","modified":"2022-03-10T18:55:13.567132+0000","last_up_change":"2022-03-10T18:55:11.106026+0000","last_in_change":"2022-03-10T18:54:33.759325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T18:55:05.021899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9d22befc-5088-4db6-ba40-146e86d78071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6809","nonce":2029997637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6810","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6811","nonce":2029997637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6815","nonce":2029997637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2029997637},{"type":"v1","addr":"172.21.15.93:6813","nonce":2029997637}]},"public_addr":"172.21.15.93:6809/2029997637","cluster_addr":"172.21.15.93:6811/2029997637","heartbeat_back_addr":"172.21.15.93:6815/2029997637","heartbeat_front_addr":"172.21.15.93:6813/2029997637","state":["exists","up"]},{"osd":1,"uuid":"a1899423-2db6-45d5-b76a-1a336454c271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6801","nonce":1882492699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6803","nonce":1882492699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6806","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6807","nonce":1882492699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6804","nonce":1882492699},{"type":"v1","addr":"172.21.15.93:6805","nonce":1882492699}]},"public_addr":"172.21.15.93:6801/1882492699","cluster_addr":"172.21.15.93:6803/1882492699","heartbeat_back_addr":"172.21.15.93:6807/1882492699","heartbeat_front_addr":"172.21.15.93:6805/1882492699","state":["exists","up"]},{"osd":2,"uuid":"964955b1-bfd5-4f0d-94ac-def1377faf36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6803","nonce":4097433165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6805","nonce":4097433165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6809","nonce":4097433165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4097433165},{"type":"v1","addr":"172.21.15.111:6807","nonce":4097433165}]},"public_addr":"172.21.15.111:6803/4097433165","cluster_addr":"172.21.15.111:6805/4097433165","heartbeat_back_addr":"172.21.15.111:6809/4097433165","heartbeat_front_addr":"172.21.15.111:6807/4097433165","state":["exists","up"]},{"osd":3,"uuid":"28521500-b718-4989-a00d-698f97d59192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6811","nonce":1593212704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6813","nonce":1593212704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6817","nonce":1593212704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1593212704},{"type":"v1","addr":"172.21.15.111:6815","nonce":1593212704}]},"public_addr":"172.21.15.111:6811/1593212704","cluster_addr":"172.21.15.111:6813/1593212704","heartbeat_back_addr":"172.21.15.111:6817/1593212704","heartbeat_front_addr":"172.21.15.111:6815/1593212704","state":["exists","up"]},{"osd":4,"uuid":"800461e8-bcf8-4517-8730-b0e34360dbc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6819","nonce":1103376889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6821","nonce":1103376889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6825","nonce":1103376889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1103376889},{"type":"v1","addr":"172.21.15.111:6823","nonce":1103376889}]},"public_addr":"172.21.15.111:6819/1103376889","cluster_addr":"172.21.15.111:6821/1103376889","heartbeat_back_addr":"172.21.15.111:6825/1103376889","heartbeat_front_addr":"172.21.15.111:6823/1103376889","state":["exists","up"]},{"osd":5,"uuid":"05f6c5ba-6061-4252-95ff-373cc1f963cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6816","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6817","nonce":2104620729}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6819","nonce":2104620729}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6823","nonce":2104620729}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6820","nonce":2104620729},{"type":"v1","addr":"172.21.15.93:6821","nonce":2104620729}]},"public_addr":"172.21.15.93:6817/2104620729","cluster_addr":"172.21.15.93:6819/2104620729","heartbeat_back_addr":"172.21.15.93:6823/2104620729","heartbeat_front_addr":"172.21.15.93:6821/2104620729","state":["exists","up"]},{"osd":6,"uuid":"c0df1a7c-6359-49c5-b10f-8495554ecfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6825","nonce":692066384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6827","nonce":692066384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6831","nonce":692066384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":692066384},{"type":"v1","addr":"172.21.15.93:6829","nonce":692066384}]},"public_addr":"172.21.15.93:6825/692066384","cluster_addr":"172.21.15.93:6827/692066384","heartbeat_back_addr":"172.21.15.93:6831/692066384","heartbeat_front_addr":"172.21.15.93:6829/692066384","state":["exists","up"]},{"osd":7,"uuid":"0a012d6b-a401-48f0-9562-4ce7a1df5190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6827","nonce":2069656674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6829","nonce":2069656674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6833","nonce":2069656674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":2069656674},{"type":"v1","addr":"172.21.15.111:6831","nonce":2069656674}]},"public_addr":"172.21.15.111:6827/2069656674","cluster_addr":"172.21.15.111:6829/2069656674","heartbeat_back_addr":"172.21.15.111:6833/2069656674","heartbeat_front_addr":"172.21.15.111:6831/2069656674","state":["exists","up"]},{"osd":8,"uuid":"508ca0d0-afad-4af4-9596-5e70f23cb384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6817","nonce":3645158221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6819","nonce":3645158221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6823","nonce":3645158221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3645158221},{"type":"v1","addr":"172.21.15.161:6821","nonce":3645158221}]},"public_addr":"172.21.15.161:6817/3645158221","cluster_addr":"172.21.15.161:6819/3645158221","heartbeat_back_addr":"172.21.15.161:6823/3645158221","heartbeat_front_addr":"172.21.15.161:6821/3645158221","state":["exists","up"]},{"osd":9,"uuid":"02df1e47-4865-428c-8ecb-da66e1cc3252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6801","nonce":3134211368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6803","nonce":3134211368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6807","nonce":3134211368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":3134211368},{"type":"v1","addr":"172.21.15.161:6805","nonce":3134211368}]},"public_addr":"172.21.15.161:6801/3134211368","cluster_addr":"172.21.15.161:6803/3134211368","heartbeat_back_addr":"172.21.15.161:6807/3134211368","heartbeat_front_addr":"172.21.15.161:6805/3134211368","state":["exists","up"]},{"osd":10,"uuid":"72166e4b-3d85-4fb5-85f2-111497256cf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6825","nonce":3615306489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6827","nonce":3615306489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6831","nonce":3615306489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3615306489},{"type":"v1","addr":"172.21.15.161:6829","nonce":3615306489}]},"public_addr":"172.21.15.161:6825/3615306489","cluster_addr":"172.21.15.161:6827/3615306489","heartbeat_back_addr":"172.21.15.161:6831/3615306489","heartbeat_front_addr":"172.21.15.161:6829/3615306489","state":["exists","up"]},{"osd":11,"uuid":"d28fc35b-19c1-4133-b927-00955809ae38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6809","nonce":2588715091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6811","nonce":2588715091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6815","nonce":2588715091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2588715091},{"type":"v1","addr":"172.21.15.161:6813","nonce":2588715091}]},"public_addr":"172.21.15.161:6809/2588715091","cluster_addr":"172.21.15.161:6811/2588715091","heartbeat_back_addr":"172.21.15.161:6815/2588715091","heartbeat_front_addr":"172.21.15.161:6813/2588715091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:54:58.907269+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:54:58.695544+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:54:57.412570+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:01.179071+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:02.393395+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:05.840639+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:09.422751+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:04.731462+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:07.963681+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:06.165232+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:09.455954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T18:55:07.881867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:0/1413700147":"2022-03-11T18:53:33.062389+0000","172.21.15.111:6800/2517481259":"2022-03-11T18:53:33.062389+0000","172.21.15.111:0/3279937313":"2022-03-11T18:53:33.062389+0000","172.21.15.111:6801/2517481259":"2022-03-11T18:53:33.062389+0000","172.21.15.111:0/3404334102":"2022-03-11T18:53:33.062389+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T18:55:23.272 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.0 flush_pg_stats 2022-03-10T18:55:23.273 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.1 flush_pg_stats 2022-03-10T18:55:23.273 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.2 flush_pg_stats 2022-03-10T18:55:23.274 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.3 flush_pg_stats 2022-03-10T18:55:23.275 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.4 flush_pg_stats 2022-03-10T18:55:23.275 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.5 flush_pg_stats 2022-03-10T18:55:23.276 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.6 flush_pg_stats 2022-03-10T18:55:23.277 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.7 flush_pg_stats 2022-03-10T18:55:23.278 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.8 flush_pg_stats 2022-03-10T18:55:23.278 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.9 flush_pg_stats 2022-03-10T18:55:23.279 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.10 flush_pg_stats 2022-03-10T18:55:23.280 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph tell osd.11 flush_pg_stats 2022-03-10T18:55:23.790 INFO:teuthology.orchestra.run.smithi093.stdout:111669149702 2022-03-10T18:55:23.790 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.10 2022-03-10T18:55:24.122 INFO:teuthology.orchestra.run.smithi093.stdout:98784247814 2022-03-10T18:55:24.123 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.5 2022-03-10T18:55:24.124 INFO:teuthology.orchestra.run.smithi093.stdout:90194313223 2022-03-10T18:55:24.125 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.4 2022-03-10T18:55:24.129 INFO:teuthology.orchestra.run.smithi093.stdout:111669149701 2022-03-10T18:55:24.129 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.6 2022-03-10T18:55:24.131 INFO:teuthology.orchestra.run.smithi093.stdout:98784247815 2022-03-10T18:55:24.131 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.9 2022-03-10T18:55:24.134 INFO:teuthology.orchestra.run.smithi093.stdout:81604378631 2022-03-10T18:55:24.135 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.1 2022-03-10T18:55:24.136 INFO:teuthology.orchestra.run.smithi093.stdout:81604378630 2022-03-10T18:55:24.137 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.0 2022-03-10T18:55:24.259 INFO:teuthology.orchestra.run.smithi093.stdout:85899345926 2022-03-10T18:55:24.259 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.3 2022-03-10T18:55:24.306 INFO:teuthology.orchestra.run.smithi093.stdout:98784247814 2022-03-10T18:55:24.306 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.7 2022-03-10T18:55:24.394 INFO:teuthology.orchestra.run.smithi093.stdout:73014444038 2022-03-10T18:55:24.395 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.2 2022-03-10T18:55:24.521 INFO:teuthology.orchestra.run.smithi093.stdout:107374182404 2022-03-10T18:55:24.521 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.8 2022-03-10T18:55:24.681 INFO:teuthology.orchestra.run.smithi093.stdout:111669149701 2022-03-10T18:55:24.753 INFO:teuthology.orchestra.run.smithi093.stdout:107374182404 2022-03-10T18:55:24.753 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.11 2022-03-10T18:55:24.857 INFO:teuthology.orchestra.run.smithi093.stdout:98784247813 2022-03-10T18:55:24.859 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149702 got 111669149701 for osd.10 2022-03-10T18:55:24.965 INFO:teuthology.orchestra.run.smithi093.stdout:111669149700 2022-03-10T18:55:24.972 INFO:teuthology.orchestra.run.smithi093.stdout:98784247813 2022-03-10T18:55:24.977 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.5 2022-03-10T18:55:25.050 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149701 got 111669149700 for osd.6 2022-03-10T18:55:25.053 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247815 got 98784247813 for osd.9 2022-03-10T18:55:25.167 INFO:teuthology.orchestra.run.smithi093.stdout:81604378629 2022-03-10T18:55:25.167 INFO:teuthology.orchestra.run.smithi093.stdout:90194313222 2022-03-10T18:55:25.219 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313223 got 90194313222 for osd.4 2022-03-10T18:55:25.219 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378630 got 81604378629 for osd.0 2022-03-10T18:55:25.225 INFO:teuthology.orchestra.run.smithi093.stdout:81604378630 2022-03-10T18:55:25.248 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378631 got 81604378630 for osd.1 2022-03-10T18:55:25.310 INFO:teuthology.orchestra.run.smithi093.stdout:98784247812 2022-03-10T18:55:25.320 INFO:teuthology.orchestra.run.smithi093.stdout:85899345925 2022-03-10T18:55:25.331 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247814 got 98784247812 for osd.7 2022-03-10T18:55:25.340 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345926 got 85899345925 for osd.3 2022-03-10T18:55:25.343 INFO:teuthology.orchestra.run.smithi093.stdout:73014444037 2022-03-10T18:55:25.360 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444038 got 73014444037 for osd.2 2022-03-10T18:55:25.404 INFO:teuthology.orchestra.run.smithi093.stdout:107374182403 2022-03-10T18:55:25.433 INFO:teuthology.orchestra.run.smithi093.stdout:107374182403 2022-03-10T18:55:25.454 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182404 got 107374182403 for osd.8 2022-03-10T18:55:25.457 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182404 got 107374182403 for osd.11 2022-03-10T18:55:25.859 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.10 2022-03-10T18:55:25.978 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.5 2022-03-10T18:55:26.051 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.6 2022-03-10T18:55:26.053 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.9 2022-03-10T18:55:26.220 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.4 2022-03-10T18:55:26.221 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.0 2022-03-10T18:55:26.249 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.1 2022-03-10T18:55:26.332 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.7 2022-03-10T18:55:26.341 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.3 2022-03-10T18:55:26.361 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.2 2022-03-10T18:55:26.454 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.8 2022-03-10T18:55:26.457 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph osd last-stat-seq osd.11 2022-03-10T18:55:26.508 INFO:teuthology.orchestra.run.smithi093.stdout:111669149702 2022-03-10T18:55:26.602 INFO:teuthology.orchestra.run.smithi093.stdout:98784247815 2022-03-10T18:55:26.683 INFO:teuthology.orchestra.run.smithi093.stdout:111669149701 2022-03-10T18:55:26.689 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149702 got 111669149702 for osd.10 2022-03-10T18:55:26.689 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:26.772 INFO:teuthology.orchestra.run.smithi093.stdout:98784247815 2022-03-10T18:55:26.828 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247814 got 98784247815 for osd.5 2022-03-10T18:55:26.829 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:26.924 INFO:teuthology.orchestra.run.smithi093.stdout:90194313223 2022-03-10T18:55:26.987 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149701 got 111669149701 for osd.6 2022-03-10T18:55:26.988 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:26.991 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247815 got 98784247815 for osd.9 2022-03-10T18:55:26.992 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:27.005 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313223 got 90194313223 for osd.4 2022-03-10T18:55:27.005 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:27.008 INFO:teuthology.orchestra.run.smithi093.stdout:81604378631 2022-03-10T18:55:27.038 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378630 got 81604378631 for osd.0 2022-03-10T18:55:27.038 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:27.112 INFO:teuthology.orchestra.run.smithi093.stdout:98784247814 2022-03-10T18:55:27.143 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.7 2022-03-10T18:55:27.143 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:27.165 INFO:teuthology.orchestra.run.smithi093.stdout:85899345927 2022-03-10T18:55:27.183 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345926 got 85899345927 for osd.3 2022-03-10T18:55:27.183 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:27.255 INFO:teuthology.orchestra.run.smithi093.stdout:73014444038 2022-03-10T18:55:27.275 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444038 got 73014444038 for osd.2 2022-03-10T18:55:27.275 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:27.311 INFO:teuthology.orchestra.run.smithi093.stdout:107374182404 2022-03-10T18:55:27.324 INFO:teuthology.orchestra.run.smithi093.stdout:81604378632 2022-03-10T18:55:27.331 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182404 got 107374182404 for osd.8 2022-03-10T18:55:27.331 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:27.348 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378631 got 81604378632 for osd.1 2022-03-10T18:55:27.348 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:27.413 INFO:teuthology.orchestra.run.smithi093.stdout:107374182404 2022-03-10T18:55:27.434 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182404 got 107374182404 for osd.11 2022-03-10T18:55:27.434 DEBUG:teuthology.parallel:result is None 2022-03-10T18:55:27.434 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-10T18:55:27.435 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph pg dump --format=json 2022-03-10T18:55:27.976 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:28.111 INFO:teuthology.orchestra.run.smithi093.stderr:dumped all 2022-03-10T18:55:28.111 INFO:teuthology.orchestra.run.smithi093.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2022-03-10T18:55:27.422099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":65996,"kb_used_data":4604,"kb_used_omap":0,"kb_used_meta":60416,"kb_avail":1124875828,"statfs":{"total":1151940427776,"available":1151872847872,"internally_reserved":0,"allocated":4714496,"data_stored":2781459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61865984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":44,"num_write_kb":1044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003283"},"pg_stats":[{"pgid":"1.0","version":"28'125","reported_seq":183,"reported_epoch":28,"state":"active+clean","last_fresh":"2022-03-10T18:55:21.712113+0000","last_change":"2022-03-10T18:55:12.746689+0000","last_active":"2022-03-10T18:55:21.712113+0000","last_peered":"2022-03-10T18:55:21.712113+0000","last_clean":"2022-03-10T18:55:21.712113+0000","last_became_active":"2022-03-10T18:55:12.744304+0000","last_became_peered":"2022-03-10T18:55:12.744304+0000","last_unstale":"2022-03-10T18:55:21.712113+0000","last_undegraded":"2022-03-10T18:55:21.712113+0000","last_fullsized":"2022-03-10T18:55:21.712113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-10T18:55:05.500096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-10T18:55:05.500096+0000","last_clean_scrub_stamp":"2022-03-10T18:55:05.500096+0000","objects_scrubbed":0,"log_size":125,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-11T19:43:22.742680+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1463904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":10,"up_from":26,"seq":111669149703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]}]},{"osd":6,"up_from":26,"seq":111669149702,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4510000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.754}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9930000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9279999999999999}]}]},{"osd":11,"up_from":25,"seq":107374182405,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6356,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738796,"statfs":{"total":95995035648,"available":95988527104,"internally_reserved":0,"allocated":798720,"data_stored":637704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.847}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4729999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2149999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0150000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.073}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7230000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.254}]}]},{"osd":8,"up_from":25,"seq":107374182405,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8620000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6850000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.966}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0059999999999998}]}]},{"osd":9,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8089999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1579999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0579999999999998}]}]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.571}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.611}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4279999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]}]},{"osd":0,"up_from":19,"seq":81604378631,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6548,"kb_used_data":724,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738604,"statfs":{"total":95995035648,"available":95988330496,"internally_reserved":0,"allocated":741376,"data_stored":577377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1680000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8180000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1280000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2050000000000001}]}]},{"osd":1,"up_from":19,"seq":81604378632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6548,"kb_used_data":724,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738604,"statfs":{"total":95995035648,"available":95988330496,"internally_reserved":0,"allocated":741376,"data_stored":577377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8130000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.792}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]}]},{"osd":3,"up_from":20,"seq":85899345927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.056}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8240000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7929999999999999}]}]},{"osd":4,"up_from":21,"seq":90194313224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5069999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.982}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0419999999999998}]}]},{"osd":5,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-10T18:55:28.113 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph pg dump --format=json 2022-03-10T18:55:28.608 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:28.610 INFO:teuthology.orchestra.run.smithi093.stderr:dumped all 2022-03-10T18:55:28.629 INFO:teuthology.orchestra.run.smithi093.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2022-03-10T18:55:27.422099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":65996,"kb_used_data":4604,"kb_used_omap":0,"kb_used_meta":60416,"kb_avail":1124875828,"statfs":{"total":1151940427776,"available":1151872847872,"internally_reserved":0,"allocated":4714496,"data_stored":2781459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61865984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":44,"num_write_kb":1044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003283"},"pg_stats":[{"pgid":"1.0","version":"28'125","reported_seq":183,"reported_epoch":28,"state":"active+clean","last_fresh":"2022-03-10T18:55:21.712113+0000","last_change":"2022-03-10T18:55:12.746689+0000","last_active":"2022-03-10T18:55:21.712113+0000","last_peered":"2022-03-10T18:55:21.712113+0000","last_clean":"2022-03-10T18:55:21.712113+0000","last_became_active":"2022-03-10T18:55:12.744304+0000","last_became_peered":"2022-03-10T18:55:12.744304+0000","last_unstale":"2022-03-10T18:55:21.712113+0000","last_undegraded":"2022-03-10T18:55:21.712113+0000","last_fullsized":"2022-03-10T18:55:21.712113+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-10T18:55:05.500096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-10T18:55:05.500096+0000","last_clean_scrub_stamp":"2022-03-10T18:55:05.500096+0000","objects_scrubbed":0,"log_size":125,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-11T19:43:22.742680+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":1463904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":10,"up_from":26,"seq":111669149703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]}]},{"osd":6,"up_from":26,"seq":111669149702,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4510000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.754}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9930000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9279999999999999}]}]},{"osd":11,"up_from":25,"seq":107374182405,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6356,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738796,"statfs":{"total":95995035648,"available":95988527104,"internally_reserved":0,"allocated":798720,"data_stored":637704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.847}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4729999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2149999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0150000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.073}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7230000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.254}]}]},{"osd":8,"up_from":25,"seq":107374182405,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8620000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6850000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.966}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0059999999999998}]}]},{"osd":9,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8089999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1579999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0579999999999998}]}]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740016,"statfs":{"total":95995035648,"available":95989776384,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.571}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.611}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4279999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]}]},{"osd":0,"up_from":19,"seq":81604378631,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6548,"kb_used_data":724,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738604,"statfs":{"total":95995035648,"available":95988330496,"internally_reserved":0,"allocated":741376,"data_stored":577377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1680000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8180000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1280000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2050000000000001}]}]},{"osd":1,"up_from":19,"seq":81604378632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6548,"kb_used_data":724,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738604,"statfs":{"total":95995035648,"available":95988330496,"internally_reserved":0,"allocated":741376,"data_stored":577377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8130000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.792}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]}]},{"osd":3,"up_from":20,"seq":85899345927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.056}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8240000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7929999999999999}]}]},{"osd":4,"up_from":21,"seq":90194313224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5069999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.982}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0419999999999998}]}]},{"osd":5,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5200,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739952,"statfs":{"total":95995035648,"available":95989710848,"internally_reserved":0,"allocated":270336,"data_stored":109889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-10T18:55:28.630 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-10T18:55:28.631 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-10T18:55:28.631 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-10T18:55:28.631 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph health --format=json 2022-03-10T18:55:29.184 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:29.184 INFO:teuthology.orchestra.run.smithi093.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-10T18:55:29.214 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-10T18:55:29.214 INFO:tasks.rook:Rook complete, yielding 2022-03-10T18:55:29.215 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-10T18:55:29.230 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph -s 2022-03-10T18:55:29.851 INFO:teuthology.orchestra.run.smithi093.stdout: cluster: 2022-03-10T18:55:29.851 INFO:teuthology.orchestra.run.smithi093.stdout: id: 2939e372-9a22-4d12-9ac6-4ccfb538e9cd 2022-03-10T18:55:29.852 INFO:teuthology.orchestra.run.smithi093.stdout: health: HEALTH_OK 2022-03-10T18:55:29.852 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:29.852 INFO:teuthology.orchestra.run.smithi093.stdout: services: 2022-03-10T18:55:29.852 INFO:teuthology.orchestra.run.smithi093.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-10T18:55:29.852 INFO:teuthology.orchestra.run.smithi093.stdout: mgr: a(active, since 116s) 2022-03-10T18:55:29.853 INFO:teuthology.orchestra.run.smithi093.stdout: osd: 12 osds: 12 up (since 18s), 12 in (since 56s) 2022-03-10T18:55:29.853 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:29.854 INFO:teuthology.orchestra.run.smithi093.stdout: data: 2022-03-10T18:55:29.854 INFO:teuthology.orchestra.run.smithi093.stdout: pools: 1 pools, 1 pgs 2022-03-10T18:55:29.854 INFO:teuthology.orchestra.run.smithi093.stdout: objects: 2 objects, 577 KiB 2022-03-10T18:55:29.854 INFO:teuthology.orchestra.run.smithi093.stdout: usage: 64 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-10T18:55:29.855 INFO:teuthology.orchestra.run.smithi093.stdout: pgs: 1 active+clean 2022-03-10T18:55:29.855 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:29.855 INFO:teuthology.orchestra.run.smithi093.stdout: io: 2022-03-10T18:55:29.855 INFO:teuthology.orchestra.run.smithi093.stdout: client: 1.5 KiB/s rd, 87 KiB/s wr, 1 op/s rd, 3 op/s wr 2022-03-10T18:55:29.856 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:55:29.876 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status 2022-03-10T18:55:30.574 INFO:teuthology.orchestra.run.smithi093.stdout:Backend: rook 2022-03-10T18:55:30.575 INFO:teuthology.orchestra.run.smithi093.stdout:Available: Yes 2022-03-10T18:55:30.597 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ps 2022-03-10T18:55:31.158 INFO:teuthology.orchestra.run.smithi093.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-10T18:55:31.158 INFO:teuthology.orchestra.run.smithi093.stdout:crashcollector.crash smithi093 running (2m) 0s ago 2m - - 2d8b0d650431 2022-03-10T18:55:31.158 INFO:teuthology.orchestra.run.smithi093.stdout:crashcollector.crash smithi111 running (62s) 0s ago 62s - - 2d8b0d650431 2022-03-10T18:55:31.159 INFO:teuthology.orchestra.run.smithi093.stdout:crashcollector.crash smithi161 running (119s) 0s ago 119s - - 2d8b0d650431 2022-03-10T18:55:31.159 INFO:teuthology.orchestra.run.smithi093.stdout:mgr.a smithi111 running (2m) 0s ago 2m - - 2d8b0d650431 2022-03-10T18:55:31.159 INFO:teuthology.orchestra.run.smithi093.stdout:mon.a smithi161 running (4m) 0s ago 5m - - 2d8b0d650431 2022-03-10T18:55:31.159 INFO:teuthology.orchestra.run.smithi093.stdout:mon.b smithi093 running (4m) 0s ago 4m - - 2d8b0d650431 2022-03-10T18:55:31.160 INFO:teuthology.orchestra.run.smithi093.stdout:mon.c smithi111 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:55:31.160 INFO:teuthology.orchestra.run.smithi093.stdout:osd.0 smithi093 running (66s) 0s ago 67s - - 2d8b0d650431 2022-03-10T18:55:31.160 INFO:teuthology.orchestra.run.smithi093.stdout:osd.1 smithi093 running (65s) 0s ago 65s - - 2d8b0d650431 2022-03-10T18:55:31.160 INFO:teuthology.orchestra.run.smithi093.stdout:osd.10 smithi161 running (49s) 0s ago 49s - - 2d8b0d650431 2022-03-10T18:55:31.161 INFO:teuthology.orchestra.run.smithi093.stdout:osd.11 smithi161 running (47s) 0s ago 48s - - 2d8b0d650431 2022-03-10T18:55:31.161 INFO:teuthology.orchestra.run.smithi093.stdout:osd.2 smithi111 running (63s) 0s ago 63s - - 2d8b0d650431 2022-03-10T18:55:31.161 INFO:teuthology.orchestra.run.smithi093.stdout:osd.3 smithi111 running (60s) 0s ago 61s - - 2d8b0d650431 2022-03-10T18:55:31.161 INFO:teuthology.orchestra.run.smithi093.stdout:osd.4 smithi111 running (59s) 0s ago 59s - - 2d8b0d650431 2022-03-10T18:55:31.161 INFO:teuthology.orchestra.run.smithi093.stdout:osd.5 smithi093 running (57s) 0s ago 58s - - 2d8b0d650431 2022-03-10T18:55:31.162 INFO:teuthology.orchestra.run.smithi093.stdout:osd.6 smithi093 running (54s) 0s ago 54s - - 2d8b0d650431 2022-03-10T18:55:31.162 INFO:teuthology.orchestra.run.smithi093.stdout:osd.7 smithi111 running (56s) 0s ago 56s - - 2d8b0d650431 2022-03-10T18:55:31.162 INFO:teuthology.orchestra.run.smithi093.stdout:osd.8 smithi161 running (53s) 0s ago 53s - - 2d8b0d650431 2022-03-10T18:55:31.162 INFO:teuthology.orchestra.run.smithi093.stdout:osd.9 smithi161 running (50s) 0s ago 51s - - 2d8b0d650431 2022-03-10T18:55:31.182 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls 2022-03-10T18:55:31.840 INFO:teuthology.orchestra.run.smithi093.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-10T18:55:31.841 INFO:teuthology.orchestra.run.smithi093.stdout:crash 3/3 0s ago 2m * 2022-03-10T18:55:31.841 INFO:teuthology.orchestra.run.smithi093.stdout:mgr 1/1 0s ago 2m count:1 2022-03-10T18:55:31.841 INFO:teuthology.orchestra.run.smithi093.stdout:mon 3/3 0s ago 5m count:3 2022-03-10T18:55:31.842 INFO:teuthology.orchestra.run.smithi093.stdout:osd 24 0s ago 67s 2022-03-10T18:55:31.842 INFO:teuthology.orchestra.run.smithi093.stdout:osd.all-available-devices 0 0s ago - * 2022-03-10T18:55:31.864 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch device ls 2022-03-10T18:55:32.435 INFO:teuthology.orchestra.run.smithi093.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-10T18:55:32.435 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T18:55:32.436 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T18:55:32.436 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T18:55:32.436 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T18:55:32.436 INFO:teuthology.orchestra.run.smithi093.stdout:smithi111 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T18:55:32.437 INFO:teuthology.orchestra.run.smithi093.stdout:smithi111 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T18:55:32.437 INFO:teuthology.orchestra.run.smithi093.stdout:smithi111 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T18:55:32.437 INFO:teuthology.orchestra.run.smithi093.stdout:smithi111 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T18:55:32.437 INFO:teuthology.orchestra.run.smithi093.stdout:smithi161 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T18:55:32.437 INFO:teuthology.orchestra.run.smithi093.stdout:smithi161 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T18:55:32.438 INFO:teuthology.orchestra.run.smithi093.stdout:smithi161 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T18:55:32.438 INFO:teuthology.orchestra.run.smithi093.stdout:smithi161 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T18:55:32.456 INFO:teuthology.run_tasks:Running task exec... 2022-03-10T18:55:32.469 INFO:teuthology.task.exec:Executing custom commands... 2022-03-10T18:55:32.470 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi093.front.sepia.ceph.com 2022-03-10T18:55:32.470 DEBUG:teuthology.orchestra.run.smithi093:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-10T18:55:32.470 DEBUG:teuthology.orchestra.run.smithi093:> toolbox() { 2022-03-10T18:55:32.471 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-10T18:55:32.471 DEBUG:teuthology.orchestra.run.smithi093:> } 2022-03-10T18:55:32.471 DEBUG:teuthology.orchestra.run.smithi093:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-10T18:55:32.472 DEBUG:teuthology.orchestra.run.smithi093:> toolbox ceph orch osd rm 0 --force 2022-03-10T18:55:32.472 DEBUG:teuthology.orchestra.run.smithi093:> removed_pv="" 2022-03-10T18:55:32.472 DEBUG:teuthology.orchestra.run.smithi093:> while [ "$removed_pv" = "" ] 2022-03-10T18:55:32.472 DEBUG:teuthology.orchestra.run.smithi093:> do 2022-03-10T18:55:32.473 DEBUG:teuthology.orchestra.run.smithi093:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-10T18:55:32.473 DEBUG:teuthology.orchestra.run.smithi093:> sleep 3s 2022-03-10T18:55:32.473 DEBUG:teuthology.orchestra.run.smithi093:> done 2022-03-10T18:55:32.474 DEBUG:teuthology.orchestra.run.smithi093:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-10T18:55:32.474 DEBUG:teuthology.orchestra.run.smithi093:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-10T18:55:32.474 DEBUG:teuthology.orchestra.run.smithi093:> toolbox ceph orch device zap $host $target_path --force 2022-03-10T18:55:32.474 DEBUG:teuthology.orchestra.run.smithi093:> zap_completion="0" 2022-03-10T18:55:32.475 DEBUG:teuthology.orchestra.run.smithi093:> while [ "$zap_completion" = "0" ] 2022-03-10T18:55:32.475 DEBUG:teuthology.orchestra.run.smithi093:> do 2022-03-10T18:55:32.475 DEBUG:teuthology.orchestra.run.smithi093:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-10T18:55:32.476 DEBUG:teuthology.orchestra.run.smithi093:> sleep 3s 2022-03-10T18:55:32.476 DEBUG:teuthology.orchestra.run.smithi093:> done 2022-03-10T18:55:32.476 DEBUG:teuthology.orchestra.run.smithi093:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-10T18:55:32.476 DEBUG:teuthology.orchestra.run.smithi093:> toolbox ceph orch apply osd --all-available-devices 2022-03-10T18:55:32.477 DEBUG:teuthology.orchestra.run.smithi093:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-10T18:55:32.477 DEBUG:teuthology.orchestra.run.smithi093:> num_osd="0" 2022-03-10T18:55:32.477 DEBUG:teuthology.orchestra.run.smithi093:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-10T18:55:32.478 DEBUG:teuthology.orchestra.run.smithi093:> do 2022-03-10T18:55:32.478 DEBUG:teuthology.orchestra.run.smithi093:> echo "waiting for osd to come back up" 2022-03-10T18:55:32.478 DEBUG:teuthology.orchestra.run.smithi093:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-10T18:55:32.478 DEBUG:teuthology.orchestra.run.smithi093:> sleep 30s 2022-03-10T18:55:32.479 DEBUG:teuthology.orchestra.run.smithi093:> done 2022-03-10T18:55:32.479 DEBUG:teuthology.orchestra.run.smithi093:> ' 2022-03-10T18:55:32.495 INFO:teuthology.orchestra.run.smithi093.stderr:++ toolbox ceph osd stat 2022-03-10T18:55:32.495 INFO:teuthology.orchestra.run.smithi093.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T18:55:32.495 INFO:teuthology.orchestra.run.smithi093.stderr:++ cut -f3 -d ' ' 2022-03-10T18:55:35.718 INFO:teuthology.orchestra.run.smithi093.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T18:55:36.167 INFO:teuthology.orchestra.run.smithi093.stderr:+ orig_num_osd=12 2022-03-10T18:55:36.167 INFO:teuthology.orchestra.run.smithi093.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-10T18:55:36.168 INFO:teuthology.orchestra.run.smithi093.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-10T18:55:36.244 INFO:teuthology.orchestra.run.smithi093.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T18:55:36.863 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:36.860961 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-10T18:55:36.865 INFO:tasks.rook.operator.smithi093.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", AllowUnsupported: true}, 2022-03-10T18:55:36.865 INFO:tasks.rook.operator.smithi093.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-10T18:55:36.866 INFO:tasks.rook.operator.smithi093.stdout:   ... // 3 identical fields 2022-03-10T18:55:36.866 INFO:tasks.rook.operator.smithi093.stdout:   Config: nil, 2022-03-10T18:55:36.866 INFO:tasks.rook.operator.smithi093.stdout:   Selection: {}, 2022-03-10T18:55:36.866 INFO:tasks.rook.operator.smithi093.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T18:55:36.866 INFO:tasks.rook.operator.smithi093.stdout:-  { 2022-03-10T18:55:36.867 INFO:tasks.rook.operator.smithi093.stdout:-  Name: "smithi093-lv-1", 2022-03-10T18:55:36.867 INFO:tasks.rook.operator.smithi093.stdout:-  Count: 1, 2022-03-10T18:55:36.867 INFO:tasks.rook.operator.smithi093.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-10T18:55:36.867 INFO:tasks.rook.operator.smithi093.stdout:-  }, 2022-03-10T18:55:36.868 INFO:tasks.rook.operator.smithi093.stdout:   {Name: "smithi093-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi093-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-10T18:55:36.868 INFO:tasks.rook.operator.smithi093.stdout:   {Name: "smithi093-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi093-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T18:55:36.869 INFO:tasks.rook.operator.smithi093.stdout:   ... // 9 identical elements 2022-03-10T18:55:36.869 INFO:tasks.rook.operator.smithi093.stdout:   }, 2022-03-10T18:55:36.869 INFO:tasks.rook.operator.smithi093.stdout:   }, 2022-03-10T18:55:36.869 INFO:tasks.rook.operator.smithi093.stdout:   Annotations: nil, 2022-03-10T18:55:36.870 INFO:tasks.rook.operator.smithi093.stdout:   Labels: nil, 2022-03-10T18:55:36.870 INFO:tasks.rook.operator.smithi093.stdout:   ... // 20 identical fields 2022-03-10T18:55:36.870 INFO:tasks.rook.operator.smithi093.stdout:  } 2022-03-10T18:55:36.870 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:36.861053 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T18:55:36.870 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:36.861367 I | operator: successfully started the controller-runtime manager 2022-03-10T18:55:36.871 INFO:tasks.rook.operator.smithi093.stdout:I0310 18:55:36.861819 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T18:55:36.910 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:36.908324 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T18:55:36.911 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:36.908344 I | operator: watching all namespaces for Ceph CRs 2022-03-10T18:55:36.911 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:36.908368 I | operator: setting up schemes 2022-03-10T18:55:36.914 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:36.910593 I | operator: setting up the controller-runtime manager 2022-03-10T18:55:37.215 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.213433 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T18:55:37.264 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.261578 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T18:55:37.265 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.261696 I | ceph-cluster-controller: successfully started 2022-03-10T18:55:37.265 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.262098 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T18:55:37.265 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.262190 I | ceph-crashcollector-controller: successfully started 2022-03-10T18:55:37.266 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.262256 I | ceph-block-pool-controller: successfully started 2022-03-10T18:55:37.266 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.262288 I | ceph-object-store-user-controller: successfully started 2022-03-10T18:55:37.266 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.262349 I | ceph-object-realm-controller: successfully started 2022-03-10T18:55:37.266 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.262372 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T18:55:37.267 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.262422 I | ceph-object-zone-controller: successfully started 2022-03-10T18:55:37.268 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.263647 I | ceph-object-controller: successfully started 2022-03-10T18:55:37.269 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.263786 I | ceph-file-controller: successfully started 2022-03-10T18:55:37.269 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.263856 I | ceph-nfs-controller: successfully started 2022-03-10T18:55:37.269 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265267 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T18:55:37.270 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265312 I | ceph-client-controller: successfully started 2022-03-10T18:55:37.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265330 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T18:55:37.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265359 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T18:55:37.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265385 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T18:55:37.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265405 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T18:55:37.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265420 I | ceph-bucket-topic: successfully started 2022-03-10T18:55:37.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265440 I | ceph-bucket-notification: successfully started 2022-03-10T18:55:37.273 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265452 I | ceph-bucket-notification: successfully started 2022-03-10T18:55:37.273 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.265469 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T18:55:37.274 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.268927 I | operator: starting the controller-runtime manager 2022-03-10T18:55:37.379 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.377039 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T18:55:37.379 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.377076 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T18:55:37.379 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.377085 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T18:55:37.386 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.383052 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T18:55:37.416 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.414732 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T18:55:37.422 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.418338 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:55:37.423 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.418440 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:55:37.423 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.418636 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T18:55:37.423 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.418663 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T18:55:37.426 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.422194 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T18:55:37.427 INFO:tasks.rook.operator.smithi093.stdout:I0310 18:55:37.422332 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T18:55:37.432 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.431238 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T18:55:37.438 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435311 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T18:55:37.438 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435332 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T18:55:37.438 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435337 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T18:55:37.438 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435341 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T18:55:37.439 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435346 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T18:55:37.439 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435352 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T18:55:37.439 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435357 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T18:55:37.439 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435362 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T18:55:37.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435366 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T18:55:37.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435372 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T18:55:37.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435376 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T18:55:37.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435382 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T18:55:37.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435388 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T18:55:37.441 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435392 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T18:55:37.441 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435421 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T18:55:37.441 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435430 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T18:55:37.441 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435489 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:55:37.442 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.435497 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:55:37.447 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:37.444909 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210... 2022-03-10T18:55:41.251 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:41.249841 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-10T18:55:41.465 INFO:teuthology.orchestra.run.smithi093.stdout:Success 2022-03-10T18:55:41.483 INFO:teuthology.orchestra.run.smithi093.stderr:+ removed_pv= 2022-03-10T18:55:41.483 INFO:teuthology.orchestra.run.smithi093.stderr:+ '[' '' = '' ']' 2022-03-10T18:55:41.483 INFO:teuthology.orchestra.run.smithi093.stderr:++ kubectl get pv 2022-03-10T18:55:41.483 INFO:teuthology.orchestra.run.smithi093.stderr:++ grep Released 2022-03-10T18:55:41.484 INFO:teuthology.orchestra.run.smithi093.stderr:++ cut -f1 -d ' ' 2022-03-10T18:55:41.567 INFO:teuthology.orchestra.run.smithi093.stderr:+ removed_pv=smithi093-lv-1 2022-03-10T18:55:41.567 INFO:teuthology.orchestra.run.smithi093.stderr:+ sleep 3s 2022-03-10T18:55:44.568 INFO:teuthology.orchestra.run.smithi093.stderr:+ '[' smithi093-lv-1 = '' ']' 2022-03-10T18:55:44.569 INFO:teuthology.orchestra.run.smithi093.stderr:++ kubectl get pv smithi093-lv-1 -o 'jsonpath={.spec.local.path}' 2022-03-10T18:55:44.897 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:44.894708 I | ceph-spec: detected ceph image version: "17.0.0-11084 quincy" 2022-03-10T18:55:44.897 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:44.894751 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T18:55:44.897 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:44.894776 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11084 quincy", pursuing 2022-03-10T18:55:44.898 INFO:teuthology.orchestra.run.smithi093.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-03-10T18:55:44.899 INFO:teuthology.orchestra.run.smithi093.stderr:++ echo smithi093-lv-1 2022-03-10T18:55:44.899 INFO:teuthology.orchestra.run.smithi093.stderr:++ cut -f1 -d - 2022-03-10T18:55:44.900 INFO:teuthology.orchestra.run.smithi093.stderr:+ host=smithi093 2022-03-10T18:55:44.901 INFO:teuthology.orchestra.run.smithi093.stderr:+ toolbox ceph orch device zap smithi093 /dev/vg_nvme/lv_1 --force 2022-03-10T18:55:44.901 INFO:teuthology.orchestra.run.smithi093.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi093 /dev/vg_nvme/lv_1 --force 2022-03-10T18:55:44.928 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:44.925636 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:55:45.060 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:45.057292 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:55:45.063 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:45.058713 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:55:45.119 INFO:teuthology.orchestra.run.smithi093.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T18:55:45.489 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:45.487300 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11084 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210" 2022-03-10T18:55:45.613 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:45.610952 I | op-mon: start running mons 2022-03-10T18:55:45.633 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/vg_nvme/lv_1 on smithi093 zapped 2022-03-10T18:55:45.656 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:45.653878 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:55:45.660 INFO:teuthology.orchestra.run.smithi093.stderr:+ zap_completion=0 2022-03-10T18:55:45.661 INFO:teuthology.orchestra.run.smithi093.stderr:+ '[' 0 = 0 ']' 2022-03-10T18:55:45.661 INFO:teuthology.orchestra.run.smithi093.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-10T18:55:45.734 INFO:teuthology.orchestra.run.smithi093.stderr:+ zap_completion= 2022-03-10T18:55:45.734 INFO:teuthology.orchestra.run.smithi093.stderr:+ sleep 3s 2022-03-10T18:55:45.794 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:45.792248 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.161:6789","172.21.15.93:6789","172.21.15.111:6789"]}] data:b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:2] 2022-03-10T18:55:45.807 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:45.805759 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:55:45.807 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:45.805926 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:55:47.490 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:47.487897 I | op-mon: targeting the mon count 3 2022-03-10T18:55:47.496 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:47.494622 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:55:47.838 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:47.836528 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:55:47.838 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:47.836549 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:55:48.615 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608115 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T18:55:48.616 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608146 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T18:55:48.616 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608154 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T18:55:48.616 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608232 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T18:55:48.617 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608239 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T18:55:48.617 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608243 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T18:55:48.617 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608247 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T18:55:48.617 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608250 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T18:55:48.618 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608273 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T18:55:48.618 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608281 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T18:55:48.618 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608286 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T18:55:48.618 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608290 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T18:55:48.619 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608294 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T18:55:48.619 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608298 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T18:55:48.619 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608302 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T18:55:48.619 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608307 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T18:55:48.620 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608311 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T18:55:48.620 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608314 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T18:55:48.620 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608318 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T18:55:48.620 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608323 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T18:55:48.620 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608327 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T18:55:48.621 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608331 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T18:55:48.621 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608335 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T18:55:48.621 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608340 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T18:55:48.621 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.608344 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T18:55:48.736 INFO:teuthology.orchestra.run.smithi093.stderr:+ '[' '' = 0 ']' 2022-03-10T18:55:48.736 INFO:teuthology.orchestra.run.smithi093.stderr:+ kubectl patch pv smithi093-lv-1 -p '{"spec":{"claimRef": null}}' 2022-03-10T18:55:48.868 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.866691 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T18:55:48.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.871348 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:55:48.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.871366 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:55:48.874 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.871370 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:55:48.874 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.871372 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:55:48.874 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.871375 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:55:48.874 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.871379 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:55:48.875 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.871383 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T18:55:48.910 INFO:teuthology.orchestra.run.smithi093.stdout:persistentvolume/smithi093-lv-1 patched 2022-03-10T18:55:48.912 INFO:teuthology.orchestra.run.smithi093.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-10T18:55:48.912 INFO:teuthology.orchestra.run.smithi093.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-10T18:55:48.917 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.915856 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:55:48.918 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:48.915877 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:55:49.121 INFO:teuthology.orchestra.run.smithi093.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T18:55:49.128 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.123240 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:55:49.129 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.123265 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:55:49.129 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.123272 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T18:55:49.173 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.171556 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T18:55:49.306 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.304352 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:55:49.306 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.304383 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:55:49.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.381871 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:55:49.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.381895 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:55:49.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.381902 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T18:55:49.447 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.446047 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:55:49.448 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.446068 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:55:49.448 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.446073 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T18:55:49.538 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:49.534757 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T18:55:50.463 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:50.461253 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:55:50.463 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:50.461273 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T18:55:50.864 INFO:teuthology.orchestra.run.smithi093.stdout:['Success'] 2022-03-10T18:55:50.867 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:50.864837 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-10T18:55:50.868 INFO:tasks.rook.operator.smithi093.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", AllowUnsupported: true}, 2022-03-10T18:55:50.868 INFO:tasks.rook.operator.smithi093.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-10T18:55:50.868 INFO:tasks.rook.operator.smithi093.stdout:   ... // 3 identical fields 2022-03-10T18:55:50.868 INFO:tasks.rook.operator.smithi093.stdout:   Config: nil, 2022-03-10T18:55:50.869 INFO:tasks.rook.operator.smithi093.stdout:   Selection: {}, 2022-03-10T18:55:50.869 INFO:tasks.rook.operator.smithi093.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T18:55:50.869 INFO:tasks.rook.operator.smithi093.stdout:   ... // 9 identical elements 2022-03-10T18:55:50.869 INFO:tasks.rook.operator.smithi093.stdout:   {Name: "smithi161-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi161-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T18:55:50.870 INFO:tasks.rook.operator.smithi093.stdout:   {Name: "smithi161-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi161-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-10T18:55:50.870 INFO:tasks.rook.operator.smithi093.stdout:+  { 2022-03-10T18:55:50.870 INFO:tasks.rook.operator.smithi093.stdout:+  Name: "smithi093-lv-1", 2022-03-10T18:55:50.870 INFO:tasks.rook.operator.smithi093.stdout:+  Count: 1, 2022-03-10T18:55:50.871 INFO:tasks.rook.operator.smithi093.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-10T18:55:50.871 INFO:tasks.rook.operator.smithi093.stdout:+  }, 2022-03-10T18:55:50.872 INFO:tasks.rook.operator.smithi093.stdout:   }, 2022-03-10T18:55:50.872 INFO:tasks.rook.operator.smithi093.stdout:   }, 2022-03-10T18:55:50.872 INFO:tasks.rook.operator.smithi093.stdout:   Annotations: nil, 2022-03-10T18:55:50.872 INFO:tasks.rook.operator.smithi093.stdout:   Labels: nil, 2022-03-10T18:55:50.872 INFO:tasks.rook.operator.smithi093.stdout:   ... // 20 identical fields 2022-03-10T18:55:50.873 INFO:tasks.rook.operator.smithi093.stdout:  } 2022-03-10T18:55:50.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:50.864914 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T18:55:50.873 INFO:tasks.rook.operator.smithi093.stdout:I0310 18:55:50.865087 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T18:55:50.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:50.865789 E | ceph-csi: failed to reconcile failed to configure ceph csi: failed to start ceph csi drivers: failed to create cephfs service "csi-cephfsplugin-metrics": failed to update service csi-cephfsplugin-metrics. Put "https://10.96.0.1:443/api/v1/namespaces/rook-ceph/services/csi-cephfsplugin-metrics": context canceled 2022-03-10T18:55:50.889 INFO:teuthology.orchestra.run.smithi093.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-10T18:55:50.991 INFO:teuthology.orchestra.run.smithi093.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-10T18:55:50.999 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:50.989583 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T18:55:50.999 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:50.989616 I | operator: watching all namespaces for Ceph CRs 2022-03-10T18:55:51.000 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:50.989661 I | operator: setting up schemes 2022-03-10T18:55:51.003 INFO:teuthology.orchestra.run.smithi093.stderr:+ num_osd=0 2022-03-10T18:55:51.004 INFO:teuthology.orchestra.run.smithi093.stderr:+ '[' 0 '!=' 12 ']' 2022-03-10T18:55:51.004 INFO:teuthology.orchestra.run.smithi093.stdout:waiting for osd to come back up 2022-03-10T18:55:51.005 INFO:teuthology.orchestra.run.smithi093.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T18:55:51.005 INFO:teuthology.orchestra.run.smithi093.stderr:++ toolbox ceph osd stat 2022-03-10T18:55:51.005 INFO:teuthology.orchestra.run.smithi093.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T18:55:51.006 INFO:teuthology.orchestra.run.smithi093.stderr:++ cut -f3 -d ' ' 2022-03-10T18:55:51.006 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.002364 I | operator: setting up the controller-runtime manager 2022-03-10T18:55:51.133 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.131341 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T18:55:51.134 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.131359 I | op-mon: checking for basic quorum with existing mons 2022-03-10T18:55:51.195 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.192983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-03-10T18:55:51.195 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.193074 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T18:55:51.216 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.213997 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:55:51.219 INFO:teuthology.orchestra.run.smithi093.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T18:55:51.244 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.241820 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-03-10 18:55:51.233482485 +0000 UTC m=+448.376425215 LastTransitionTime:2022-03-10 18:55:51.233482398 +0000 UTC m=+448.376425141}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:55:51.244 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.241860 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210... 2022-03-10T18:55:51.306 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.304165 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T18:55:51.368 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.365988 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T18:55:51.372 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366071 I | ceph-cluster-controller: successfully started 2022-03-10T18:55:51.372 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366140 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T18:55:51.372 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366169 I | ceph-crashcollector-controller: successfully started 2022-03-10T18:55:51.372 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366198 I | ceph-block-pool-controller: successfully started 2022-03-10T18:55:51.373 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366224 I | ceph-object-store-user-controller: successfully started 2022-03-10T18:55:51.374 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366248 I | ceph-object-realm-controller: successfully started 2022-03-10T18:55:51.374 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366266 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T18:55:51.374 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366282 I | ceph-object-zone-controller: successfully started 2022-03-10T18:55:51.375 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366859 I | ceph-object-controller: successfully started 2022-03-10T18:55:51.375 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.366966 I | ceph-file-controller: successfully started 2022-03-10T18:55:51.375 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367052 I | ceph-nfs-controller: successfully started 2022-03-10T18:55:51.376 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367096 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T18:55:51.376 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367131 I | ceph-client-controller: successfully started 2022-03-10T18:55:51.376 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367197 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T18:55:51.376 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367240 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T18:55:51.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367268 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T18:55:51.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367291 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T18:55:51.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367345 I | ceph-bucket-topic: successfully started 2022-03-10T18:55:51.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367368 I | ceph-bucket-notification: successfully started 2022-03-10T18:55:51.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367383 I | ceph-bucket-notification: successfully started 2022-03-10T18:55:51.378 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.367397 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T18:55:51.378 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.368652 I | operator: starting the controller-runtime manager 2022-03-10T18:55:51.479 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.477360 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T18:55:51.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.477413 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T18:55:51.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.477493 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T18:55:51.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.477511 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T18:55:51.547 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.545482 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T18:55:51.553 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.548837 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:55:51.614 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.612302 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:55:51.614 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.612411 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T18:55:51.614 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.612457 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T18:55:51.618 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.613940 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T18:55:51.618 INFO:tasks.rook.operator.smithi093.stdout:I0310 18:55:51.614009 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T18:55:51.878 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:51.876434 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T18:55:52.003 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.000908 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210... 2022-03-10T18:55:52.352 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349576 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T18:55:52.353 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349600 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T18:55:52.353 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349606 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T18:55:52.353 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349612 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T18:55:52.353 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349617 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T18:55:52.354 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349621 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T18:55:52.354 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349624 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T18:55:52.354 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349628 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T18:55:52.354 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349631 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T18:55:52.355 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349635 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T18:55:52.356 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349638 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T18:55:52.356 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349642 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T18:55:52.356 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349646 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T18:55:52.356 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349649 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T18:55:52.357 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349652 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T18:55:52.357 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349658 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T18:55:52.357 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349738 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:55:52.357 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.349753 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:55:52.555 INFO:teuthology.orchestra.run.smithi093.stderr:+ num_osd=11 2022-03-10T18:55:52.555 INFO:teuthology.orchestra.run.smithi093.stderr:+ sleep 30s 2022-03-10T18:55:52.570 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.566182 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-10T18:55:52.733 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.730984 I | op-k8sutil: Removing previous job rook-ceph-csi-detect-version to start a new one 2022-03-10T18:55:52.747 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.745648 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T18:55:52.802 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:52.799891 I | op-k8sutil: batch job rook-ceph-csi-detect-version still exists 2022-03-10T18:55:55.753 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:55.751260 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T18:55:55.804 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:55.802481 I | op-k8sutil: batch job rook-ceph-csi-detect-version deleted 2022-03-10T18:55:58.760 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:55:58.757380 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T18:56:02.018 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:02.015611 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-10T18:56:03.270 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265809 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T18:56:03.270 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265855 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T18:56:03.270 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265873 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T18:56:03.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265885 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T18:56:03.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265896 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T18:56:03.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265908 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T18:56:03.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265921 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T18:56:03.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265932 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T18:56:03.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265946 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T18:56:03.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265959 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T18:56:03.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265971 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T18:56:03.273 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265983 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T18:56:03.273 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.265995 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T18:56:03.273 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266006 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T18:56:03.273 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266017 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T18:56:03.273 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266030 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T18:56:03.274 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266042 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T18:56:03.274 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266053 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T18:56:03.274 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266064 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T18:56:03.275 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266075 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T18:56:03.275 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266086 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T18:56:03.275 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266097 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T18:56:03.275 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266106 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T18:56:03.276 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266120 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T18:56:03.276 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.266130 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T18:56:03.654 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.652442 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T18:56:03.660 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.658474 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:56:03.660 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.658499 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:56:03.661 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.658503 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:56:03.661 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.658506 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:56:03.661 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.658508 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:56:03.661 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.658511 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:56:03.662 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.658514 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T18:56:03.866 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.863909 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:56:03.866 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.863930 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:56:03.867 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.863934 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T18:56:03.919 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:03.917395 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T18:56:04.295 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.292802 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T18:56:04.296 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.292832 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T18:56:04.296 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.292841 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T18:56:04.428 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.425775 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T18:56:04.428 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.425799 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T18:56:04.429 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.425808 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T18:56:04.477 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.475638 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T18:56:04.685 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.683755 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T18:56:04.690 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.688193 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T18:56:04.690 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.688215 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T18:56:04.696 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:04.692637 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T18:56:08.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.257870 I | ceph-spec: detected ceph image version: "17.0.0-11084 quincy" 2022-03-10T18:56:08.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.257901 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T18:56:08.261 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.257912 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11084 quincy", pursuing 2022-03-10T18:56:08.305 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.302810 I | ceph-spec: detected ceph image version: "17.0.0-11084 quincy" 2022-03-10T18:56:08.305 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.302883 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T18:56:08.306 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.302919 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11084 quincy", pursuing 2022-03-10T18:56:08.311 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.305376 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:56:08.340 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.338759 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:56:08.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.374356 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:56:08.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.374504 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:56:08.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.374880 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:56:08.381 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.375310 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:56:08.813 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.811291 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11084 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210" 2022-03-10T18:56:08.814 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.811537 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11084 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210" 2022-03-10T18:56:08.933 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.931180 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-10 18:56:08.828915399 +0000 UTC m=+465.971858144 LastTransitionTime:2022-03-10 18:56:08.82891532 +0000 UTC m=+465.971858069}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:56:08.983 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:08.981590 I | op-mon: start running mons 2022-03-10T18:56:09.025 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:09.023649 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mons LastHeartbeatTime:2022-03-10 18:56:08.978276774 +0000 UTC m=+466.121219516 LastTransitionTime:2022-03-10 18:56:08.97827668 +0000 UTC m=+466.121219430}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:56:09.030 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:09.024867 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:56:10.026 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:10.023895 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.93:6789","172.21.15.111:6789","172.21.15.161:6789"]}] data:a=172.21.15.161:6789,b=172.21.15.93:6789,c=172.21.15.111:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:2] 2022-03-10T18:56:10.624 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:10.622438 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:56:10.625 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:10.622743 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:56:12.226 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:12.223806 I | op-mon: targeting the mon count 3 2022-03-10T18:56:12.235 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:12.232928 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:56:12.573 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:12.571385 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:56:12.574 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:12.571423 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:56:12.941 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:12.938748 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:56:12.941 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:12.938781 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:56:13.277 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:13.274714 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:56:13.277 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:13.274751 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:56:13.681 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:13.678785 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:56:13.681 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:13.678822 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T18:56:14.016 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.013948 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T18:56:14.017 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.013968 I | op-mon: checking for basic quorum with existing mons 2022-03-10T18:56:14.017 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.013973 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:56:14.017 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.013979 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:56:14.017 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.013982 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:56:14.055 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.053391 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.161:6789","172.21.15.93:6789","172.21.15.111:6789"]}] data:b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:2] 2022-03-10T18:56:14.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.062492 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:56:14.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.062668 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:56:14.082 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.079877 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T18:56:14.091 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.089543 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T18:56:14.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.089658 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:56:14.432 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.429873 I | op-mon: mons running: [b c a] 2022-03-10T18:56:14.861 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.858709 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:56:14.887 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.885279 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T18:56:14.894 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.892736 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T18:56:14.894 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:14.892759 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:56:15.033 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:15.031570 I | op-mon: mons running: [b c a] 2022-03-10T18:56:15.449 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:15.446912 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:56:15.461 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:15.459509 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T18:56:15.469 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:15.467467 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T18:56:15.470 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:15.467487 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:56:15.634 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:15.632356 I | op-mon: mons running: [b c a] 2022-03-10T18:56:16.038 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:16.035955 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:56:16.039 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:16.035984 I | op-mon: mons created: 3 2022-03-10T18:56:16.452 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:16.450441 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:56:16.514 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:16.506791 I | op-mon: mons running: [b c a] 2022-03-10T18:56:16.941 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:16.939181 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:56:16.942 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:16.939236 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T18:56:16.942 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:16.939300 I | op-mon: start running mons 2022-03-10T18:56:16.946 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:16.943203 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:56:17.359 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:17.357069 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T18:56:17.626 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:17.623882 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.93:6789","172.21.15.111:6789","172.21.15.161:6789"]}] data:b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:2] 2022-03-10T18:56:17.787 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:17.784720 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T18:56:18.215 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:18.213364 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T18:56:18.223 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:18.222014 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:56:18.226 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:18.222245 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:56:20.826 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:20.823995 I | op-mon: targeting the mon count 3 2022-03-10T18:56:20.835 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:20.833014 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:56:21.167 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:21.164595 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T18:56:21.167 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:21.164626 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:56:21.425 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:21.423282 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T18:56:21.426 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:21.423301 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T18:56:21.587 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:21.584886 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T18:56:21.587 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:21.584921 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:56:21.875 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:21.873244 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T18:56:21.927 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:21.924889 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T18:56:21.927 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:21.924916 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:56:22.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.269707 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T18:56:22.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.269742 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T18:56:22.302 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.300439 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T18:56:22.302 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.300486 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T18:56:22.303 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.300497 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T18:56:22.557 INFO:teuthology.orchestra.run.smithi093.stderr:+ '[' 11 '!=' 12 ']' 2022-03-10T18:56:22.557 INFO:teuthology.orchestra.run.smithi093.stdout:waiting for osd to come back up 2022-03-10T18:56:22.558 INFO:teuthology.orchestra.run.smithi093.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T18:56:22.558 INFO:teuthology.orchestra.run.smithi093.stderr:++ toolbox ceph osd stat 2022-03-10T18:56:22.558 INFO:teuthology.orchestra.run.smithi093.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T18:56:22.559 INFO:teuthology.orchestra.run.smithi093.stderr:++ cut -f3 -d ' ' 2022-03-10T18:56:22.619 INFO:teuthology.orchestra.run.smithi093.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T18:56:22.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.684952 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T18:56:22.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.684984 I | op-mon: checking for basic quorum with existing mons 2022-03-10T18:56:22.688 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.684995 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:56:22.688 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.685009 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:56:22.688 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.685016 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-10T18:56:22.711 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.708796 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.93:6789","172.21.15.111:6789","172.21.15.161:6789"]}] data:b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 mapping:{"node":{"a":{"Name":"smithi161","Hostname":"smithi161","Address":"172.21.15.161"},"b":{"Name":"smithi093","Hostname":"smithi093","Address":"172.21.15.93"},"c":{"Name":"smithi111","Hostname":"smithi111","Address":"172.21.15.111"}}} maxMonId:2] 2022-03-10T18:56:22.796 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:22.793954 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T18:56:23.096 INFO:teuthology.orchestra.run.smithi093.stderr:+ num_osd=11 2022-03-10T18:56:23.097 INFO:teuthology.orchestra.run.smithi093.stderr:+ sleep 30s 2022-03-10T18:56:23.225 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:23.223134 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T18:56:23.229 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:23.223433 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T18:56:23.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:23.437476 I | op-mgr: start running mgr 2022-03-10T18:56:23.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:23.437512 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T18:56:23.888 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:23.886643 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T18:56:23.898 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:23.895860 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T18:56:23.898 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:23.895886 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:56:24.834 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:24.831925 I | op-mon: mons running: [b c a] 2022-03-10T18:56:25.185 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:25.183694 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T18:56:25.193 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:25.191401 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T18:56:25.196 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:25.194070 I | op-mgr: setting services to point to mgr "a" 2022-03-10T18:56:25.300 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:25.298432 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:56:25.369 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:25.367002 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T18:56:25.378 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:25.377003 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T18:56:25.379 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:25.377029 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:56:26.434 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.431879 I | op-mon: mons running: [b c a] 2022-03-10T18:56:26.631 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.623011 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T18:56:26.632 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.623371 I | op-mgr: successful modules: balancer 2022-03-10T18:56:26.665 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.663400 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T18:56:26.666 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.663428 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T18:56:26.705 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.703503 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T18:56:26.827 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.823904 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi093-lv-2" 2022-03-10T18:56:26.828 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.823954 I | op-osd: OSD PVC "smithi093-lv-2-data-0556qk" already exists 2022-03-10T18:56:26.828 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.823974 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi093-lv-3" 2022-03-10T18:56:26.828 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.823993 I | op-osd: OSD PVC "smithi093-lv-3-data-0q2rqj" already exists 2022-03-10T18:56:26.828 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824007 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi093-lv-4" 2022-03-10T18:56:26.829 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824027 I | op-osd: OSD PVC "smithi093-lv-4-data-0hwwsd" already exists 2022-03-10T18:56:26.829 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824042 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi111-lv-1" 2022-03-10T18:56:26.829 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824058 I | op-osd: OSD PVC "smithi111-lv-1-data-0z2jls" already exists 2022-03-10T18:56:26.829 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824070 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi111-lv-2" 2022-03-10T18:56:26.830 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824089 I | op-osd: OSD PVC "smithi111-lv-2-data-0rqjrb" already exists 2022-03-10T18:56:26.830 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824106 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi111-lv-3" 2022-03-10T18:56:26.830 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824124 I | op-osd: OSD PVC "smithi111-lv-3-data-02m2mz" already exists 2022-03-10T18:56:26.830 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824137 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi111-lv-4" 2022-03-10T18:56:26.830 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824155 I | op-osd: OSD PVC "smithi111-lv-4-data-0cfp2j" already exists 2022-03-10T18:56:26.831 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824167 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi161-lv-1" 2022-03-10T18:56:26.831 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824184 I | op-osd: OSD PVC "smithi161-lv-1-data-0vstrh" already exists 2022-03-10T18:56:26.831 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824197 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi161-lv-2" 2022-03-10T18:56:26.831 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824214 I | op-osd: OSD PVC "smithi161-lv-2-data-0b6h5v" already exists 2022-03-10T18:56:26.832 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824260 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi161-lv-3" 2022-03-10T18:56:26.832 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824281 I | op-osd: OSD PVC "smithi161-lv-3-data-0dw8c4" already exists 2022-03-10T18:56:26.832 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824294 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi161-lv-4" 2022-03-10T18:56:26.832 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824311 I | op-osd: OSD PVC "smithi161-lv-4-data-0m9dpg" already exists 2022-03-10T18:56:26.832 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.824324 I | op-osd: creating 1 new PVCs for device set "smithi093-lv-1" 2022-03-10T18:56:26.955 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.953085 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:56:26.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.959633 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T18:56:26.969 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.967457 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T18:56:26.969 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:26.967480 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:56:27.025 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.023143 I | op-osd: successfully provisioned PVC "smithi093-lv-1-data-0ssg9m" 2022-03-10T18:56:27.089 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085456 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-2-data-0556qk" 2022-03-10T18:56:27.089 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085475 I | op-osd: skipping OSD prepare job creation for PVC "smithi093-lv-2-data-0556qk" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.090 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085480 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:56:27.090 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085483 I | op-osd: skipping OSD prepare job creation for PVC "smithi093-lv-3-data-0q2rqj" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.090 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085486 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:56:27.090 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085489 I | op-osd: skipping OSD prepare job creation for PVC "smithi093-lv-4-data-0hwwsd" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.091 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085492 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-1-data-0z2jls" 2022-03-10T18:56:27.091 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085495 I | op-osd: skipping OSD prepare job creation for PVC "smithi111-lv-1-data-0z2jls" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.091 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085498 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:56:27.091 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085501 I | op-osd: skipping OSD prepare job creation for PVC "smithi111-lv-2-data-0rqjrb" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.091 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085503 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-3-data-02m2mz" 2022-03-10T18:56:27.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085506 I | op-osd: skipping OSD prepare job creation for PVC "smithi111-lv-3-data-02m2mz" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085509 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:56:27.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085511 I | op-osd: skipping OSD prepare job creation for PVC "smithi111-lv-4-data-0cfp2j" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085514 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-1-data-0vstrh" 2022-03-10T18:56:27.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085517 I | op-osd: skipping OSD prepare job creation for PVC "smithi161-lv-1-data-0vstrh" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.093 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085520 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:56:27.093 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085522 I | op-osd: skipping OSD prepare job creation for PVC "smithi161-lv-2-data-0b6h5v" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.093 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085525 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:56:27.093 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085528 I | op-osd: skipping OSD prepare job creation for PVC "smithi161-lv-3-data-0dw8c4" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.093 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085532 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:56:27.094 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085537 I | op-osd: skipping OSD prepare job creation for PVC "smithi161-lv-4-data-0m9dpg" because OSD daemon using the PVC already exists 2022-03-10T18:56:27.094 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.085542 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-1-data-0ssg9m" 2022-03-10T18:56:27.434 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.432345 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T18:56:27.860 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.858451 I | op-osd: started OSD provisioning job for PVC "smithi093-lv-1-data-0ssg9m" 2022-03-10T18:56:27.861 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.858479 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T18:56:27.861 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:27.858486 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T18:56:28.048 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.046070 I | op-mon: mons running: [b c a] 2022-03-10T18:56:28.224 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.222325 I | op-osd: OSD orchestration status for PVC smithi093-lv-1-data-0ssg9m is "starting" 2022-03-10T18:56:28.326 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.323663 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:28.333 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.330772 I | op-osd: updating OSD 1 on PVC "smithi093-lv-2-data-0556qk" 2022-03-10T18:56:28.333 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.330799 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-2-data-0556qk" 2022-03-10T18:56:28.490 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.487736 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:56:28.490 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.487766 I | op-mon: mons created: 3 2022-03-10T18:56:28.530 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.528301 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:28.552 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.549799 I | op-osd: updating OSD 10 on PVC "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:56:28.552 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.549818 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:56:28.592 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.587677 I | op-mgr: successful modules: dashboard 2022-03-10T18:56:28.598 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.596790 I | op-mgr: successful modules: prometheus 2022-03-10T18:56:28.895 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.893037 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T18:56:28.936 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:28.934232 I | op-mgr: successful modules: orchestrator modules 2022-03-10T18:56:29.119 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:29.117087 I | op-mon: mons running: [b c a] 2022-03-10T18:56:29.205 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:29.202588 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:29.327 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:29.324806 I | op-osd: updating OSD 11 on PVC "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:56:29.327 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:29.324836 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:56:29.609 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:29.607134 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T18:56:29.667 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:29.607174 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T18:56:29.851 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:29.849454 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:29.862 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:29.859723 I | op-osd: updating OSD 2 on PVC "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:56:29.862 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:29.859763 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:56:30.153 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:30.151354 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:30.158 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:30.156529 I | op-osd: updating OSD 3 on PVC "smithi111-lv-1-data-0z2jls" 2022-03-10T18:56:30.158 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:30.156561 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-1-data-0z2jls" 2022-03-10T18:56:30.296 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:30.294604 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T18:56:30.435 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:30.432957 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:30.532 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:30.530289 I | op-osd: updating OSD 4 on PVC "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:56:30.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:30.530312 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:56:31.173 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:31.171227 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T18:56:31.324 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:31.290323 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:31.543 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:31.540908 I | op-osd: updating OSD 5 on PVC "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:56:31.543 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:31.540954 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:56:31.880 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:31.878657 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T18:56:32.053 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:32.051816 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:32.129 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:32.127755 I | op-osd: updating OSD 6 on PVC "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:56:32.130 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:32.127774 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:56:32.863 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:32.861337 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:32.982 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:32.980176 I | op-osd: updating OSD 7 on PVC "smithi111-lv-3-data-02m2mz" 2022-03-10T18:56:32.982 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:32.980206 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-3-data-02m2mz" 2022-03-10T18:56:33.046 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:33.044008 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T18:56:33.047 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:33.044029 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T18:56:33.457 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:33.455324 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T18:56:33.635 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:33.633564 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:33.732 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:33.729905 I | op-osd: updating OSD 8 on PVC "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:56:33.732 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:33.729942 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:56:33.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:33.870255 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T18:56:33.874 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:33.870289 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T18:56:33.874 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:33.870295 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T18:56:34.797 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:34.795364 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T18:56:34.882 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:34.879930 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:34.965 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:34.962606 I | op-osd: updating OSD 9 on PVC "smithi161-lv-1-data-0vstrh" 2022-03-10T18:56:34.965 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:34.962642 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-1-data-0vstrh" 2022-03-10T18:56:34.977 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:34.974662 I | op-mgr: start running mgr 2022-03-10T18:56:34.977 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:34.974697 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T18:56:35.022 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:35.020305 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 9 on PVC "smithi161-lv-1-data-0vstrh" LastHeartbeatTime:2022-03-10 18:56:34.971432743 +0000 UTC m=+492.114375475 LastTransitionTime:2022-03-10 18:56:34.971432627 +0000 UTC m=+492.114375381}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:56:35.649 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:35.647027 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T18:56:35.656 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:35.654080 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T18:56:35.732 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:35.730102 I | op-mgr: setting services to point to mgr "a" 2022-03-10T18:56:36.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.531461 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T18:56:36.534 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.531518 I | op-mgr: successful modules: balancer 2022-03-10T18:56:36.543 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.541542 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T18:56:36.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.541566 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T18:56:36.634 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.624890 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T18:56:36.648 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645553 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi093-lv-2" 2022-03-10T18:56:36.649 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645593 I | op-osd: OSD PVC "smithi093-lv-2-data-0556qk" already exists 2022-03-10T18:56:36.649 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645605 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi093-lv-3" 2022-03-10T18:56:36.649 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645615 I | op-osd: OSD PVC "smithi093-lv-3-data-0q2rqj" already exists 2022-03-10T18:56:36.649 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645622 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi093-lv-4" 2022-03-10T18:56:36.650 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645630 I | op-osd: OSD PVC "smithi093-lv-4-data-0hwwsd" already exists 2022-03-10T18:56:36.650 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645637 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi111-lv-1" 2022-03-10T18:56:36.650 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645645 I | op-osd: OSD PVC "smithi111-lv-1-data-0z2jls" already exists 2022-03-10T18:56:36.651 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645652 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi111-lv-2" 2022-03-10T18:56:36.651 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645659 I | op-osd: OSD PVC "smithi111-lv-2-data-0rqjrb" already exists 2022-03-10T18:56:36.651 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645728 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi111-lv-3" 2022-03-10T18:56:36.651 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645741 I | op-osd: OSD PVC "smithi111-lv-3-data-02m2mz" already exists 2022-03-10T18:56:36.652 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645747 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi111-lv-4" 2022-03-10T18:56:36.652 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645756 I | op-osd: OSD PVC "smithi111-lv-4-data-0cfp2j" already exists 2022-03-10T18:56:36.652 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645764 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi161-lv-1" 2022-03-10T18:56:36.652 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645773 I | op-osd: OSD PVC "smithi161-lv-1-data-0vstrh" already exists 2022-03-10T18:56:36.652 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645786 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi161-lv-2" 2022-03-10T18:56:36.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645800 I | op-osd: OSD PVC "smithi161-lv-2-data-0b6h5v" already exists 2022-03-10T18:56:36.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645810 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi161-lv-3" 2022-03-10T18:56:36.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645819 I | op-osd: OSD PVC "smithi161-lv-3-data-0dw8c4" already exists 2022-03-10T18:56:36.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645825 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi161-lv-4" 2022-03-10T18:56:36.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645832 I | op-osd: OSD PVC "smithi161-lv-4-data-0m9dpg" already exists 2022-03-10T18:56:36.654 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645838 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi093-lv-1" 2022-03-10T18:56:36.654 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.645846 I | op-osd: OSD PVC "smithi093-lv-1-data-0ssg9m" already exists 2022-03-10T18:56:36.853 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849737 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-2-data-0556qk" 2022-03-10T18:56:36.853 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849755 I | op-osd: skipping OSD prepare job creation for PVC "smithi093-lv-2-data-0556qk" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.854 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849759 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:56:36.854 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849762 I | op-osd: skipping OSD prepare job creation for PVC "smithi093-lv-3-data-0q2rqj" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.854 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849765 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:56:36.854 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849767 I | op-osd: skipping OSD prepare job creation for PVC "smithi093-lv-4-data-0hwwsd" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.855 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849770 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-1-data-0z2jls" 2022-03-10T18:56:36.855 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849773 I | op-osd: skipping OSD prepare job creation for PVC "smithi111-lv-1-data-0z2jls" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.856 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849778 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:56:36.856 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849782 I | op-osd: skipping OSD prepare job creation for PVC "smithi111-lv-2-data-0rqjrb" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.856 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849786 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-3-data-02m2mz" 2022-03-10T18:56:36.856 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849790 I | op-osd: skipping OSD prepare job creation for PVC "smithi111-lv-3-data-02m2mz" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.857 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849794 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:56:36.857 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849798 I | op-osd: skipping OSD prepare job creation for PVC "smithi111-lv-4-data-0cfp2j" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.857 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849802 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-1-data-0vstrh" 2022-03-10T18:56:36.857 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849806 I | op-osd: skipping OSD prepare job creation for PVC "smithi161-lv-1-data-0vstrh" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.858 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849811 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:56:36.858 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849815 I | op-osd: skipping OSD prepare job creation for PVC "smithi161-lv-2-data-0b6h5v" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.859 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849819 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:56:36.859 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849823 I | op-osd: skipping OSD prepare job creation for PVC "smithi161-lv-3-data-0dw8c4" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.859 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849828 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:56:36.859 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849832 I | op-osd: skipping OSD prepare job creation for PVC "smithi161-lv-4-data-0m9dpg" because OSD daemon using the PVC already exists 2022-03-10T18:56:36.860 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:36.849836 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-1-data-0ssg9m" 2022-03-10T18:56:37.163 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.160512 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi093-lv-1-data-0ssg9m. Status={Conditions:[] StartTime:2022-03-10 18:56:27 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-10T18:56:37.163 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.160537 I | op-osd: started OSD provisioning job for PVC "smithi093-lv-1-data-0ssg9m" 2022-03-10T18:56:37.163 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.160546 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T18:56:37.164 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.160551 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T18:56:37.165 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.163126 I | op-osd: OSD orchestration status for PVC smithi093-lv-1-data-0ssg9m is "starting" 2022-03-10T18:56:37.266 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.264687 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:37.338 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.336471 I | op-osd: updating OSD 1 on PVC "smithi093-lv-2-data-0556qk" 2022-03-10T18:56:37.339 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.336492 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-2-data-0556qk" 2022-03-10T18:56:37.531 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.529048 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:37.616 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.613395 I | op-osd: updating OSD 10 on PVC "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:56:37.616 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:37.613420 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-3-data-0dw8c4" 2022-03-10T18:56:38.031 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.029168 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:38.037 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.035435 I | op-osd: updating OSD 11 on PVC "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:56:38.037 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.035476 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-4-data-0m9dpg" 2022-03-10T18:56:38.239 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.237634 I | op-osd: OSD orchestration status for node smithi093-lv-1-data-0ssg9m is "orchestrating" 2022-03-10T18:56:38.538 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.536257 I | op-osd: OSD orchestration status for node smithi093-lv-1-data-0ssg9m is "orchestrating" 2022-03-10T18:56:38.618 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.615767 I | op-osd: OSD orchestration status for PVC smithi093-lv-1-data-0ssg9m is "orchestrating" 2022-03-10T18:56:38.618 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.615911 I | op-osd: OSD orchestration status for PVC smithi093-lv-1-data-0ssg9m is "orchestrating" 2022-03-10T18:56:38.718 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.716701 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:38.767 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.765038 I | op-osd: updating OSD 2 on PVC "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:56:38.770 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.765112 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-2-data-0rqjrb" 2022-03-10T18:56:38.783 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.781715 I | op-mgr: successful modules: dashboard 2022-03-10T18:56:38.786 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.782595 I | op-mgr: successful modules: prometheus 2022-03-10T18:56:38.790 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:38.788296 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T18:56:39.435 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:39.433360 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:39.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:39.530896 I | op-osd: updating OSD 3 on PVC "smithi111-lv-1-data-0z2jls" 2022-03-10T18:56:39.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:39.530929 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-1-data-0z2jls" 2022-03-10T18:56:40.237 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:40.235288 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:40.332 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:40.330503 I | op-osd: updating OSD 4 on PVC "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:56:40.333 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:40.330523 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-4-data-0cfp2j" 2022-03-10T18:56:40.376 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:40.374507 I | op-mgr: successful modules: orchestrator modules 2022-03-10T18:56:41.039 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:41.037387 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:41.141 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:41.139030 I | op-osd: updating OSD 5 on PVC "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:56:41.141 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:41.139053 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-4-data-0hwwsd" 2022-03-10T18:56:41.853 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:41.851251 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:41.934 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:41.932075 I | op-osd: updating OSD 6 on PVC "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:56:41.935 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:41.932116 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-3-data-0q2rqj" 2022-03-10T18:56:42.640 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:42.637654 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:42.742 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:42.740505 I | op-osd: updating OSD 7 on PVC "smithi111-lv-3-data-02m2mz" 2022-03-10T18:56:42.743 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:42.740538 I | op-osd: OSD will have its main bluestore block on "smithi111-lv-3-data-02m2mz" 2022-03-10T18:56:43.439 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:43.437083 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:43.532 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:43.530037 I | op-osd: updating OSD 8 on PVC "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:56:43.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:43.530066 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-2-data-0b6h5v" 2022-03-10T18:56:44.239 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:44.236496 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T18:56:44.332 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:44.330007 I | op-osd: updating OSD 9 on PVC "smithi161-lv-1-data-0vstrh" 2022-03-10T18:56:44.332 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:44.330044 I | op-osd: OSD will have its main bluestore block on "smithi161-lv-1-data-0vstrh" 2022-03-10T18:56:44.596 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:44.593446 I | op-osd: OSD orchestration status for PVC smithi093-lv-1-data-0ssg9m is "completed" 2022-03-10T18:56:44.596 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:44.593487 I | op-osd: creating OSD 0 on PVC "smithi093-lv-1-data-0ssg9m" 2022-03-10T18:56:44.596 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:44.593510 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-1-data-0ssg9m" 2022-03-10T18:56:45.205 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:45.202462 I | op-osd: OSD orchestration status for PVC smithi093-lv-1-data-0ssg9m is "completed" 2022-03-10T18:56:45.209 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:45.207004 I | op-osd: creating OSD 0 on PVC "smithi093-lv-1-data-0ssg9m" 2022-03-10T18:56:45.213 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:45.211002 I | op-osd: OSD will have its main bluestore block on "smithi093-lv-1-data-0ssg9m" 2022-03-10T18:56:45.470 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:45.468091 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 0 on PVC "smithi093-lv-1-data-0ssg9m" LastHeartbeatTime:2022-03-10 18:56:45.427574882 +0000 UTC m=+502.570517622 LastTransitionTime:2022-03-10 18:56:45.427574712 +0000 UTC m=+502.570517489}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:56:45.765 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:45.762819 E | op-osd: failed to create OSD 0 on PVC "smithi093-lv-1-data-0ssg9m": failed to create deployment for OSD 0 on PVC "smithi093-lv-1-data-0ssg9m": deployments.apps "rook-ceph-osd-0" already exists 2022-03-10T18:56:45.779 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:45.777004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-10T18:56:45.779 INFO:tasks.rook.operator.smithi093.stdout:failed to create OSD 0 on PVC "smithi093-lv-1-data-0ssg9m": failed to create deployment for OSD 0 on PVC "smithi093-lv-1-data-0ssg9m": deployments.apps "rook-ceph-osd-0" already exists 2022-03-10T18:56:46.196 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:46.193136 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T18:56:46.196 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:56:46.193154 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T18:56:53.098 INFO:teuthology.orchestra.run.smithi093.stderr:+ '[' 11 '!=' 12 ']' 2022-03-10T18:56:53.099 INFO:teuthology.orchestra.run.smithi093.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T18:56:53.099 INFO:teuthology.orchestra.run.smithi093.stdout:waiting for osd to come back up 2022-03-10T18:56:53.100 INFO:teuthology.orchestra.run.smithi093.stderr:++ toolbox ceph osd stat 2022-03-10T18:56:53.100 INFO:teuthology.orchestra.run.smithi093.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T18:56:53.100 INFO:teuthology.orchestra.run.smithi093.stderr:++ cut -f3 -d ' ' 2022-03-10T18:56:53.167 INFO:teuthology.orchestra.run.smithi093.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T18:56:53.899 INFO:teuthology.orchestra.run.smithi093.stderr:+ num_osd=11 2022-03-10T18:56:53.900 INFO:teuthology.orchestra.run.smithi093.stderr:+ sleep 30s 2022-03-10T18:57:23.901 INFO:teuthology.orchestra.run.smithi093.stderr:+ '[' 11 '!=' 12 ']' 2022-03-10T18:57:23.902 INFO:teuthology.orchestra.run.smithi093.stdout:waiting for osd to come back up 2022-03-10T18:57:23.903 INFO:teuthology.orchestra.run.smithi093.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T18:57:23.903 INFO:teuthology.orchestra.run.smithi093.stderr:++ toolbox ceph osd stat 2022-03-10T18:57:23.903 INFO:teuthology.orchestra.run.smithi093.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T18:57:23.904 INFO:teuthology.orchestra.run.smithi093.stderr:++ cut -f3 -d ' ' 2022-03-10T18:57:23.976 INFO:teuthology.orchestra.run.smithi093.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T18:57:24.468 INFO:teuthology.orchestra.run.smithi093.stderr:+ num_osd=12 2022-03-10T18:57:24.469 INFO:teuthology.orchestra.run.smithi093.stderr:+ sleep 30s 2022-03-10T18:57:54.470 INFO:teuthology.orchestra.run.smithi093.stderr:+ '[' 12 '!=' 12 ']' 2022-03-10T18:57:54.472 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-10T18:57:54.483 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch status 2022-03-10T18:57:55.271 INFO:teuthology.orchestra.run.smithi093.stdout:Backend: rook 2022-03-10T18:57:55.271 INFO:teuthology.orchestra.run.smithi093.stdout:Available: Yes 2022-03-10T18:57:55.296 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ps 2022-03-10T18:57:55.827 INFO:teuthology.orchestra.run.smithi093.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-10T18:57:55.828 INFO:teuthology.orchestra.run.smithi093.stdout:crashcollector.crash smithi093 running (4m) 0s ago 4m - - 2d8b0d650431 2022-03-10T18:57:55.828 INFO:teuthology.orchestra.run.smithi093.stdout:crashcollector.crash smithi111 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.828 INFO:teuthology.orchestra.run.smithi093.stdout:crashcollector.crash smithi161 running (4m) 0s ago 4m - - 2d8b0d650431 2022-03-10T18:57:55.828 INFO:teuthology.orchestra.run.smithi093.stdout:mgr.a smithi111 running (4m) 0s ago 4m - - 2d8b0d650431 2022-03-10T18:57:55.829 INFO:teuthology.orchestra.run.smithi093.stdout:mon.a smithi161 running (6m) 0s ago 7m - - 2d8b0d650431 2022-03-10T18:57:55.829 INFO:teuthology.orchestra.run.smithi093.stdout:mon.b smithi093 running (6m) 0s ago 6m - - 2d8b0d650431 2022-03-10T18:57:55.829 INFO:teuthology.orchestra.run.smithi093.stdout:mon.c smithi111 running (5m) 0s ago 5m - - 2d8b0d650431 2022-03-10T18:57:55.829 INFO:teuthology.orchestra.run.smithi093.stdout:osd.0 smithi093 running (70s) 0s ago 70s - - 2d8b0d650431 2022-03-10T18:57:55.830 INFO:teuthology.orchestra.run.smithi093.stdout:osd.1 smithi093 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.830 INFO:teuthology.orchestra.run.smithi093.stdout:osd.10 smithi161 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.830 INFO:teuthology.orchestra.run.smithi093.stdout:osd.11 smithi161 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.830 INFO:teuthology.orchestra.run.smithi093.stdout:osd.2 smithi111 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.831 INFO:teuthology.orchestra.run.smithi093.stdout:osd.3 smithi111 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.831 INFO:teuthology.orchestra.run.smithi093.stdout:osd.4 smithi111 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.831 INFO:teuthology.orchestra.run.smithi093.stdout:osd.5 smithi093 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.831 INFO:teuthology.orchestra.run.smithi093.stdout:osd.6 smithi093 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.831 INFO:teuthology.orchestra.run.smithi093.stdout:osd.7 smithi111 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.832 INFO:teuthology.orchestra.run.smithi093.stdout:osd.8 smithi161 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.832 INFO:teuthology.orchestra.run.smithi093.stdout:osd.9 smithi161 running (3m) 0s ago 3m - - 2d8b0d650431 2022-03-10T18:57:55.849 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls 2022-03-10T18:57:56.524 INFO:teuthology.orchestra.run.smithi093.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-10T18:57:56.534 INFO:teuthology.orchestra.run.smithi093.stdout:crash 3/3 0s ago 4m * 2022-03-10T18:57:56.534 INFO:teuthology.orchestra.run.smithi093.stdout:mgr 1/1 0s ago 4m count:1 2022-03-10T18:57:56.535 INFO:teuthology.orchestra.run.smithi093.stdout:mon 3/3 0s ago 7m count:3 2022-03-10T18:57:56.535 INFO:teuthology.orchestra.run.smithi093.stdout:osd 24 0s ago 3m 2022-03-10T18:57:56.535 INFO:teuthology.orchestra.run.smithi093.stdout:osd.all-available-devices 0 0s ago - * 2022-03-10T18:57:56.549 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch host ls 2022-03-10T18:57:57.084 INFO:teuthology.orchestra.run.smithi093.stdout:HOST ADDR LABELS STATUS 2022-03-10T18:57:57.085 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 172.21.15.93/smithi093 2022-03-10T18:57:57.085 INFO:teuthology.orchestra.run.smithi093.stdout:smithi111 172.21.15.111/smithi111 2022-03-10T18:57:57.085 INFO:teuthology.orchestra.run.smithi093.stdout:smithi161 172.21.15.161/smithi161 2022-03-10T18:57:57.086 INFO:teuthology.orchestra.run.smithi093.stdout:3 hosts in cluster 2022-03-10T18:57:57.104 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch device ls 2022-03-10T18:57:57.669 INFO:teuthology.orchestra.run.smithi093.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-10T18:57:57.670 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T18:57:57.670 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T18:57:57.670 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T18:57:57.671 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T18:57:57.671 INFO:teuthology.orchestra.run.smithi093.stdout:smithi111 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T18:57:57.671 INFO:teuthology.orchestra.run.smithi093.stdout:smithi111 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T18:57:57.671 INFO:teuthology.orchestra.run.smithi093.stdout:smithi111 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T18:57:57.672 INFO:teuthology.orchestra.run.smithi093.stdout:smithi111 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T18:57:57.672 INFO:teuthology.orchestra.run.smithi093.stdout:smithi161 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T18:57:57.672 INFO:teuthology.orchestra.run.smithi093.stdout:smithi161 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T18:57:57.672 INFO:teuthology.orchestra.run.smithi093.stdout:smithi161 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T18:57:57.672 INFO:teuthology.orchestra.run.smithi093.stdout:smithi161 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T18:57:57.690 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch apply rgw foo 2022-03-10T18:57:58.259 INFO:teuthology.orchestra.run.smithi093.stdout:Created 2022-03-10T18:57:58.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:57:58.258318 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-10T18:57:58.280 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch apply mds foo 2022-03-10T18:57:58.342 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:57:58.339964 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:57:58.343 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:57:58.340012 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210... 2022-03-10T18:57:58.860 INFO:teuthology.orchestra.run.smithi093.stdout:Created 2022-03-10T18:57:58.861 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:57:58.858422 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-10T18:57:58.883 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch apply rbd-mirror 2022-03-10T18:57:58.939 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:57:58.937373 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:57:58.940 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:57:58.937439 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210... 2022-03-10T18:57:59.816 INFO:teuthology.orchestra.run.smithi093.stdout:Success 2022-03-10T18:57:59.840 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch apply nfs foo --port 12777 2022-03-10T18:57:59.850 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:57:59.847632 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:57:59.850 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:57:59.847674 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210... 2022-03-10T18:58:04.288 INFO:teuthology.orchestra.run.smithi093.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-10T18:58:04.289 INFO:teuthology.orchestra.run.smithi093.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-10T18:58:04.309 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-10T18:58:04.320 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:58:05.375 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:58:05.375 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:16.000000Z", "last_refresh": "2022-03-10T18:58:05.137602Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:05.137602Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:58:05.137602Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:58:05.137602Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:58:05.137602Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:58:05.137602Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-10T18:58:05.137602Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:05.137602Z", "running": 0, "size": 1}}] 2022-03-10T18:58:05.397 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch rm mds.foo 2022-03-10T18:58:05.639 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:05.635979 I | ceph-spec: detected ceph image version: "17.0.0-11084 quincy" 2022-03-10T18:58:06.315 INFO:teuthology.orchestra.run.smithi093.stdout:Removed cephfilesystems/foo 2022-03-10T18:58:06.341 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch rm rbd-mirror 2022-03-10T18:58:06.690 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:06.686665 I | ceph-spec: detected ceph image version: "17.0.0-11084 quincy" 2022-03-10T18:58:07.770 INFO:teuthology.orchestra.run.smithi093.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-10T18:58:07.794 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch rm rgw.foo 2022-03-10T18:58:08.441 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:08.438961 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-10T18:58:08.656 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:08.653727 I | ceph-spec: detected ceph image version: "17.0.0-11084 quincy" 2022-03-10T18:58:08.925 INFO:teuthology.orchestra.run.smithi093.stdout:Removed cephobjectstores/foo 2022-03-10T18:58:08.947 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:58:08.978 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:08.976415 I | ceph-object-controller: reconciling object store deployments 2022-03-10T18:58:09.046 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:09.040283 I | ceph-object-controller: ceph object store gateway service running at 10.111.144.92 2022-03-10T18:58:09.046 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:09.040323 I | ceph-object-controller: reconciling object store pools 2022-03-10T18:58:09.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:09.076227 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-10T18:58:09.098 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:09.096177 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-10T18:58:09.099 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:09.096203 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-10T18:58:09.099 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:09.096212 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-10T18:58:09.403 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:09.400896 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T18:58:09.660 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:58:09.661 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:16.000000Z", "last_refresh": "2022-03-10T18:58:09.495663Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:09.495663Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:58:09.495663Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:58:09.495663Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:58:09.495663Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:58:09.495663Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:09.495663Z", "running": 0, "size": 1}}] 2022-03-10T18:58:10.335 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:10.332311 I | op-mds: setting mds config flags 2022-03-10T18:58:10.335 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:10.332347 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T18:58:10.495 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:10.492442 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-10T18:58:10.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:10.650935 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-10T18:58:10.879 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:10.876734 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi161": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:58:11.069 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:11.065956 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T18:58:11.152 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:11.149964 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-10T18:58:11.582 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:11.577013 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi093": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:58:13.740 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:13.737344 I | op-mds: setting mds config flags 2022-03-10T18:58:13.740 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:13.737375 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T18:58:16.298 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:16.294245 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T18:58:17.369 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:17.366742 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-10T18:58:18.506 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:18.504043 I | ceph-file-controller: creating filesystem "foo" 2022-03-10T18:58:19.685 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:58:20.475 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:20.472300 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-10T18:58:20.477 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:20.473652 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-10T18:58:20.959 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:20.956891 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-10T18:58:20.967 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:20.964845 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-10T18:58:21.720 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:58:21.720 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:16.000000Z", "last_refresh": "2022-03-10T18:58:21.254423Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:21.254423Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:58:21.254423Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:58:21.254423Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:58:21.254423Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:58:21.254423Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:21.254423Z", "running": 0, "size": 1}}] 2022-03-10T18:58:23.606 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:23.603648 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-10T18:58:23.695 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:23.692498 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:24.482 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:24.480068 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-10T18:58:27.826 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:27.823347 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:27.815+0000 7f4a4dfd5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:27.826 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:27.815+0000 7f4a4dfd5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:27.827 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:27.815+0000 7f4a4dfd5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:27.827 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:27.859 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:27.856929 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:28.701 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:28.698734 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:28.686+0000 7fddbc5bb380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:28.702 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:28.686+0000 7fddbc5bb380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:28.702 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:28.686+0000 7fddbc5bb380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:28.702 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:28.737 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:28.735244 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:28.944 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:28.941164 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-10T18:58:29.380 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:29.377086 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:29.366+0000 7fc7dc1a8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:29.380 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:29.366+0000 7fc7dc1a8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:29.381 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:29.366+0000 7fc7dc1a8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:29.381 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:29.395 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:29.392918 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:29.617 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:29.614659 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-10T18:58:30.141 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:30.138409 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:30.130+0000 7fea4c6f5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:30.142 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:30.130+0000 7fea4c6f5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:30.142 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:30.130+0000 7fea4c6f5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:30.142 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:30.179 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:30.176310 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:30.743 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:30.740002 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:30.730+0000 7f20627f2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:30.743 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:30.730+0000 7f20627f2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:30.743 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:30.730+0000 7f20627f2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:30.744 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:30.762 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:30.759407 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:31.502 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:31.498907 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:31.486+0000 7ff6b0d13380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:31.502 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:31.486+0000 7ff6b0d13380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:31.502 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:31.486+0000 7ff6b0d13380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:31.502 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:31.520 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:31.517738 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:31.744 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:58:31.988 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:31.985622 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-10T18:58:31.988 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:31.985642 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-10T18:58:32.148 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:32.145670 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:32.134+0000 7f9b5967a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:32.149 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:32.134+0000 7f9b5967a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:32.149 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:32.134+0000 7f9b5967a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:32.149 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:32.161 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:32.159309 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:32.561 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:58:32.561 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T18:58:32.266216Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:32.266216Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:58:32.266216Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:58:32.266216Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:58:32.266216Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:58:32.266216Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:32.266216Z", "running": 0, "size": 1}}] 2022-03-10T18:58:32.751 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:32.748564 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:32.738+0000 7f113a7d7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:32.751 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:32.738+0000 7f113a7d7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:32.752 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:32.738+0000 7f113a7d7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:32.752 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:32.793 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:32.791088 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:32.988 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:32.985284 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:32.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:32.986803 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-10T18:58:33.382 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:33.379028 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:33.370+0000 7f665b066380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:33.382 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:33.370+0000 7f665b066380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:33.382 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:33.370+0000 7f665b066380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:33.382 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:33.798 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:33.795283 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-10T18:58:34.308 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:34.305245 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-10T18:58:34.308 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:34.305270 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-10T18:58:34.308 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:34.305276 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-10T18:58:34.918 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:34.914937 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-10T18:58:35.249 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:35.246928 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-10T18:58:35.766 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:35.763619 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-10T18:58:35.767 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:35.763647 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-10T18:58:35.767 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:35.763655 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-10T18:58:35.953 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:35.950459 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:36.430 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:36.427820 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-10T18:58:36.431 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:36.427852 I | ceph-file-controller: downing filesystem "foo" 2022-03-10T18:58:36.606 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:36.603255 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:36.594+0000 7f727a3b5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:36.607 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:36.594+0000 7f727a3b5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:36.607 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:36.594+0000 7f727a3b5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:36.607 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:37.278 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:37.275801 I | ceph-file-controller: downed filesystem "foo" 2022-03-10T18:58:41.639 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:41.637183 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-10T18:58:41.640 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:41.637206 I | cephclient: purging pool "foo-metadata" (id=8) 2022-03-10T18:58:41.736 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:41.728649 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:42.583 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:58:42.823 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:42.820140 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:42.810+0000 7fd2b9c89380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:42.823 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:42.810+0000 7fd2b9c89380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:42.825 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:42.810+0000 7fd2b9c89380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:42.825 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:44.034 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:44.031702 I | cephclient: purge completed for pool "foo-metadata" 2022-03-10T18:58:44.755 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:44.752439 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-10T18:58:44.755 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:44.752464 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-10T18:58:46.481 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:58:46.481 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T18:58:46.234922Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:46.234922Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:58:46.234922Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:58:46.234922Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:58:46.234922Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:58:46.234922Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:46.234922Z", "running": 0, "size": 1}}] 2022-03-10T18:58:47.513 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:47.510093 I | cephclient: purge completed for pool "foo-data0" 2022-03-10T18:58:47.513 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:47.510156 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-10T18:58:47.539 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:47.537040 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-10T18:58:47.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:47.541336 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-10T18:58:47.552 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:47.550197 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-10T18:58:47.560 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:47.558366 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-10T18:58:50.583 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:50.580891 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi093": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T18:58:53.069 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:53.066918 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:58:55.201 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:58:55.198105 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:58:55.186+0000 7f0aa425b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:58:55.201 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:55.186+0000 7f0aa425b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:58:55.201 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:58:55.186+0000 7f0aa425b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:58:55.202 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:58:56.508 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:58:57.221 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:58:57.221 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T18:58:57.047651Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:58:57.047651Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:58:57.047651Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:58:57.047651Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:58:57.047651Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:58:57.047651Z", "running": 0, "size": 1}}] 2022-03-10T18:59:07.241 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:59:07.957 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:59:07.957 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T18:59:07.764356Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:59:07.764356Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:59:07.764356Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:59:07.764356Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:59:07.764356Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:59:07.764356Z", "running": 0, "size": 1}}] 2022-03-10T18:59:15.700 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:59:15.697093 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:59:16.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:59:16.374385 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:59:16.361+0000 7f389c1b1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:59:16.378 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:59:16.361+0000 7f389c1b1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:59:16.378 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:59:16.361+0000 7f389c1b1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:59:16.379 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T18:59:18.004 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:59:18.772 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:59:18.773 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T18:59:18.574278Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:59:18.574278Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:59:18.574278Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:59:18.574278Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:59:18.574278Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:59:18.574278Z", "running": 0, "size": 1}}] 2022-03-10T18:59:28.792 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:59:29.518 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:59:29.518 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T18:59:29.341227Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:59:29.341227Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:59:29.341227Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:59:29.341227Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:59:29.341227Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:59:29.341227Z", "running": 0, "size": 1}}] 2022-03-10T18:59:39.540 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:59:40.271 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:59:40.271 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T18:59:40.095373Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:59:40.095373Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:59:40.095373Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:59:40.095373Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:59:40.095373Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:59:40.095373Z", "running": 0, "size": 1}}] 2022-03-10T18:59:50.293 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T18:59:51.001 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T18:59:51.001 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T18:59:50.823876Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T18:59:50.823876Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T18:59:50.823876Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T18:59:50.823876Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T18:59:50.823876Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T18:59:50.823876Z", "running": 0, "size": 1}}] 2022-03-10T18:59:57.344 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:59:57.341048 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T18:59:58.096 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 18:59:58.092613 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T18:59:58.084+0000 7f4bcb5b2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T18:59:58.096 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:59:58.084+0000 7f4bcb5b2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T18:59:58.096 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T18:59:58.084+0000 7f4bcb5b2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T18:59:58.097 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:00:01.023 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:00:01.718 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:00:01.718 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:00:01.550292Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:00:01.550292Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:00:01.550292Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:00:01.550292Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:00:01.550292Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:00:01.550292Z", "running": 0, "size": 1}}] 2022-03-10T19:00:11.739 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:00:12.514 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:00:12.514 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:00:12.315225Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:00:12.315225Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:00:12.315225Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:00:12.315225Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:00:12.315225Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:00:12.315225Z", "running": 0, "size": 1}}] 2022-03-10T19:00:15.678 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:00:15.675011 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi161": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T19:00:22.534 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:00:23.243 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:00:23.243 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:00:23.068647Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:00:23.068647Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:00:23.068647Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:00:23.068647Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:00:23.068647Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:00:23.068647Z", "running": 0, "size": 1}}] 2022-03-10T19:00:33.265 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:00:33.984 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:00:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:00:33.816314Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:00:33.816314Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:00:33.816314Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:00:33.816314Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:00:33.816314Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:00:33.816314Z", "running": 0, "size": 1}}] 2022-03-10T19:00:44.009 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:00:44.783 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:00:44.784 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:00:44.561123Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:00:44.561123Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:00:44.561123Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:00:44.561123Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:00:44.561123Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:00:44.561123Z", "running": 0, "size": 1}}] 2022-03-10T19:00:54.808 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:00:55.504 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:00:55.504 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:00:55.326948Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:00:55.326948Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:00:55.326948Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:00:55.326948Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:00:55.326948Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:00:55.326948Z", "running": 0, "size": 1}}] 2022-03-10T19:01:05.529 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:01:06.289 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:01:06.290 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:01:06.112120Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:01:06.112120Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:01:06.112120Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:01:06.112120Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:01:06.112120Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:01:06.112120Z", "running": 0, "size": 1}}] 2022-03-10T19:01:16.320 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:01:17.232 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:01:17.232 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:01:16.822162Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:01:16.822162Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:01:16.822162Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:01:16.822162Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:01:16.822162Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:01:16.822162Z", "running": 0, "size": 1}}] 2022-03-10T19:01:20.022 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:01:20.018812 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:01:20.668 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:01:20.663566 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:01:20.650+0000 7fa8fca5e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:01:20.668 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:01:20.650+0000 7fa8fca5e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:01:20.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:01:20.650+0000 7fa8fca5e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:01:20.669 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:01:27.256 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:01:27.949 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:01:27.950 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:01:27.778659Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:01:27.778659Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:01:27.778659Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:01:27.778659Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:01:27.778659Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:01:27.778659Z", "running": 0, "size": 1}}] 2022-03-10T19:01:37.996 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:01:38.787 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:01:38.787 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:01:38.510804Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:01:38.510804Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:01:38.510804Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:01:38.510804Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:01:38.510804Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:01:38.510804Z", "running": 0, "size": 1}}] 2022-03-10T19:01:48.806 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:01:49.549 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:01:49.550 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:01:49.367719Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:01:49.367719Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:01:49.367719Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:01:49.367719Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:01:49.367719Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:01:49.367719Z", "running": 0, "size": 1}}] 2022-03-10T19:01:59.572 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:02:00.337 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:02:00.338 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:02:00.150794Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:02:00.150794Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:02:00.150794Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:02:00.150794Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:02:00.150794Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:02:00.150794Z", "running": 0, "size": 1}}] 2022-03-10T19:02:10.358 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:02:11.045 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:02:11.045 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:02:10.844620Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:02:10.844620Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:02:10.844620Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:02:10.844620Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:02:10.844620Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:02:10.844620Z", "running": 0, "size": 1}}] 2022-03-10T19:02:21.073 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:02:21.793 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:02:21.794 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:02:21.578306Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:02:21.578306Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:02:21.578306Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:02:21.578306Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:02:21.578306Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:02:21.578306Z", "running": 0, "size": 1}}] 2022-03-10T19:02:31.814 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:02:32.509 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:02:32.509 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:02:32.320316Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:02:32.320316Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:02:32.320316Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:02:32.320316Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:02:32.320316Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:02:32.320316Z", "running": 0, "size": 1}}] 2022-03-10T19:02:42.529 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:02:43.236 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:02:43.237 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:02:43.044376Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:02:43.044376Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:02:43.044376Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:02:43.044376Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:02:43.044376Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:02:43.044376Z", "running": 0, "size": 1}}] 2022-03-10T19:02:53.256 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:02:54.038 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:02:54.038 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:02:53.866884Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:02:53.866884Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:02:53.866884Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:02:53.866884Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:02:53.866884Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:02:53.866884Z", "running": 0, "size": 1}}] 2022-03-10T19:03:04.059 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:03:04.737 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:03:04.738 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:03:04.569217Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:03:04.569217Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:03:04.569217Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:03:04.569217Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:03:04.569217Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:03:04.569217Z", "running": 0, "size": 1}}] 2022-03-10T19:03:14.763 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:03:15.529 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:03:15.530 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:03:15.287439Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:03:15.287439Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:03:15.287439Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:03:15.287439Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:03:15.287439Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:03:15.287439Z", "running": 0, "size": 1}}] 2022-03-10T19:03:25.555 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:03:26.248 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:03:26.249 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:03:26.086156Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:03:26.086156Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:03:26.086156Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:03:26.086156Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:03:26.086156Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:03:26.086156Z", "running": 0, "size": 1}}] 2022-03-10T19:03:36.274 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:03:37.026 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:03:37.027 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:03:36.787220Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:03:36.787220Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:03:36.787220Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:03:36.787220Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:03:36.787220Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:03:36.787220Z", "running": 0, "size": 1}}] 2022-03-10T19:03:47.047 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:03:47.767 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:03:47.768 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:03:47.569869Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:03:47.569869Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:03:47.569869Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:03:47.569869Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:03:47.569869Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:03:47.569869Z", "running": 0, "size": 1}}] 2022-03-10T19:03:57.788 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:03:58.443 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:03:58.444 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:03:58.285648Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:03:58.285648Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:03:58.285648Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:03:58.285648Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:03:58.285648Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:03:58.285648Z", "running": 0, "size": 1}}] 2022-03-10T19:04:04.516 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:04:04.511864 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:04:05.156 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:04:05.151963 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:04:05.138+0000 7f5324a9f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:04:05.157 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:04:05.138+0000 7f5324a9f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:04:05.157 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:04:05.138+0000 7f5324a9f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:04:05.157 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:04:08.466 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:04:09.199 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:04:09.200 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:04:09.044021Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:04:09.044021Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:04:09.044021Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:04:09.044021Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:04:09.044021Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:04:09.044021Z", "running": 0, "size": 1}}] 2022-03-10T19:04:19.220 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:04:19.952 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:04:19.952 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:04:19.723683Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:04:19.723683Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:04:19.723683Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:04:19.723683Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:04:19.723683Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:04:19.723683Z", "running": 0, "size": 1}}] 2022-03-10T19:04:29.976 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:04:30.725 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:04:31.022 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:04:30.462734Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:04:30.462734Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:04:30.462734Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:04:30.462734Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:04:30.462734Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:04:30.462734Z", "running": 0, "size": 1}}] 2022-03-10T19:04:41.026 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:04:41.707 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:04:41.707 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:04:41.518865Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:04:41.518865Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:04:41.518865Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:04:41.518865Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:04:41.518865Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:04:41.518865Z", "running": 0, "size": 1}}] 2022-03-10T19:04:51.739 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:04:52.463 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:04:52.464 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:04:52.272514Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:04:52.272514Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:04:52.272514Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:04:52.272514Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:04:52.272514Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:04:52.272514Z", "running": 0, "size": 1}}] 2022-03-10T19:05:02.487 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:05:03.284 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:05:03.284 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:05:03.100965Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:05:03.100965Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:05:03.100965Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:05:03.100965Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:05:03.100965Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:05:03.100965Z", "running": 0, "size": 1}}] 2022-03-10T19:05:13.306 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:05:14.047 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:05:14.047 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:05:13.849786Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:05:13.849786Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:05:13.849786Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:05:13.849786Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:05:13.849786Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:05:13.849786Z", "running": 0, "size": 1}}] 2022-03-10T19:05:24.072 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:05:24.776 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:05:24.776 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:05:24.605987Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:05:24.605987Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:05:24.605987Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:05:24.605987Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:05:24.605987Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:05:24.605987Z", "running": 0, "size": 1}}] 2022-03-10T19:05:34.800 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:05:35.508 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:05:35.508 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:05:35.315856Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:05:35.315856Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:05:35.315856Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:05:35.315856Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:05:35.315856Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:05:35.315856Z", "running": 0, "size": 1}}] 2022-03-10T19:05:45.528 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:05:46.198 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:05:46.199 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:05:46.027085Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:05:46.027085Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:05:46.027085Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:05:46.027085Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:05:46.027085Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:05:46.027085Z", "running": 0, "size": 1}}] 2022-03-10T19:05:56.218 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:05:56.899 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:05:56.900 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:05:56.726468Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:05:56.726468Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:05:56.726468Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:05:56.726468Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:05:56.726468Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:05:56.726468Z", "running": 0, "size": 1}}] 2022-03-10T19:06:06.923 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:06:07.616 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:06:07.616 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:06:07.441186Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:06:07.441186Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:06:07.441186Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:06:07.441186Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:06:07.441186Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:06:07.441186Z", "running": 0, "size": 1}}] 2022-03-10T19:06:17.642 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:06:18.746 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:06:19.303 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:06:18.166530Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:06:18.166530Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:06:18.166530Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:06:18.166530Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:06:18.166530Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:06:18.166530Z", "running": 0, "size": 1}}] 2022-03-10T19:06:29.307 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:06:30.043 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:06:30.044 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:06:29.846897Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:06:29.846897Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:06:29.846897Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:06:29.846897Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:06:29.846897Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:06:29.846897Z", "running": 0, "size": 1}}] 2022-03-10T19:06:40.068 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:06:40.832 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:06:40.832 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:06:40.596269Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:06:40.596269Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:06:40.596269Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:06:40.596269Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:06:40.596269Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:06:40.596269Z", "running": 0, "size": 1}}] 2022-03-10T19:06:50.856 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:06:51.631 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:06:51.632 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:06:51.458167Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:06:51.458167Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:06:51.458167Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:06:51.458167Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:06:51.458167Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:06:51.458167Z", "running": 0, "size": 1}}] 2022-03-10T19:07:01.655 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:07:02.336 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:07:02.336 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:07:02.165608Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:07:02.165608Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:07:02.165608Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:07:02.165608Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:07:02.165608Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:07:02.165608Z", "running": 0, "size": 1}}] 2022-03-10T19:07:12.360 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:07:13.108 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:07:13.108 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:07:12.906505Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:07:12.906505Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:07:12.906505Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:07:12.906505Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:07:12.906505Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:07:12.906505Z", "running": 0, "size": 1}}] 2022-03-10T19:07:23.128 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:07:23.810 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:07:23.811 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:07:23.649721Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:07:23.649721Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:07:23.649721Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:07:23.649721Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:07:23.649721Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:07:23.649721Z", "running": 0, "size": 1}}] 2022-03-10T19:07:33.831 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:07:34.560 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:07:34.561 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:07:34.394782Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:07:34.394782Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:07:34.394782Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:07:34.394782Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:07:34.394782Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:07:34.394782Z", "running": 0, "size": 1}}] 2022-03-10T19:07:44.581 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:07:45.326 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:07:45.326 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:07:45.122557Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:07:45.122557Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:07:45.122557Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:07:45.122557Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:07:45.122557Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:07:45.122557Z", "running": 0, "size": 1}}] 2022-03-10T19:07:55.350 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:07:56.044 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:07:56.045 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:07:55.840145Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:07:55.840145Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:07:55.840145Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:07:55.840145Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:07:55.840145Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:07:55.840145Z", "running": 0, "size": 1}}] 2022-03-10T19:08:06.072 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:08:06.782 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:08:06.783 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:08:06.604412Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:08:06.604412Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:08:06.604412Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:08:06.604412Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:08:06.604412Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:08:06.604412Z", "running": 0, "size": 1}}] 2022-03-10T19:08:16.805 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:08:17.597 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:08:17.597 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:08:17.386305Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:08:17.386305Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:08:17.386305Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:08:17.386305Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:08:17.386305Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:08:17.386305Z", "running": 0, "size": 1}}] 2022-03-10T19:08:27.620 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:08:28.283 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:08:28.283 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:08:28.107989Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:08:28.107989Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:08:28.107989Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:08:28.107989Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:08:28.107989Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:08:28.107989Z", "running": 0, "size": 1}}] 2022-03-10T19:08:38.307 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:08:39.080 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:08:39.080 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:08:38.913589Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:08:38.913589Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:08:38.913589Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:08:38.913589Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:08:38.913589Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:08:38.913589Z", "running": 0, "size": 1}}] 2022-03-10T19:08:49.100 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:08:52.614 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:08:52.614 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:08:52.454550Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:08:52.454550Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:08:52.454550Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:08:52.454550Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:08:52.454550Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:08:52.454550Z", "running": 0, "size": 1}}] 2022-03-10T19:09:02.635 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:09:03.441 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:09:03.442 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:09:03.209360Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:09:03.209360Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:09:03.209360Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:09:03.209360Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:09:03.209360Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:09:03.209360Z", "running": 0, "size": 1}}] 2022-03-10T19:09:13.469 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:09:14.204 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:09:14.204 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:09:13.986949Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:09:13.986949Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:09:13.986949Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:09:13.986949Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:09:13.986949Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:09:13.986949Z", "running": 0, "size": 1}}] 2022-03-10T19:09:24.228 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:09:24.935 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:09:24.935 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:09:24.774337Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:09:24.774337Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:09:24.774337Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:09:24.774337Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:09:24.774337Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:09:24.774337Z", "running": 0, "size": 1}}] 2022-03-10T19:09:32.844 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:09:32.839666 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:09:33.483 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:09:33.478940 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:09:33.467+0000 7f9c4ce35380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:09:33.483 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:09:33.467+0000 7f9c4ce35380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:09:33.484 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:09:33.467+0000 7f9c4ce35380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:09:33.484 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:09:34.961 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:09:35.713 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:09:35.713 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:09:35.456533Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:09:35.456533Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:09:35.456533Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:09:35.456533Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:09:35.456533Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:09:35.456533Z", "running": 0, "size": 1}}] 2022-03-10T19:09:45.738 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:09:46.468 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:09:46.469 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:09:46.286580Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:09:46.286580Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:09:46.286580Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:09:46.286580Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:09:46.286580Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:09:46.286580Z", "running": 0, "size": 1}}] 2022-03-10T19:09:56.517 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:09:57.243 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:09:57.244 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:09:57.072972Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:09:57.072972Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:09:57.072972Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:09:57.072972Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:09:57.072972Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:09:57.072972Z", "running": 0, "size": 1}}] 2022-03-10T19:10:07.269 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:10:07.972 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:10:07.973 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:10:07.798716Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:10:07.798716Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:10:07.798716Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:10:07.798716Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:10:07.798716Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:10:07.798716Z", "running": 0, "size": 1}}] 2022-03-10T19:10:18.002 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:10:18.749 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:10:18.750 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:10:18.565679Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:10:18.565679Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:10:18.565679Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:10:18.565679Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:10:18.565679Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:10:18.565679Z", "running": 0, "size": 1}}] 2022-03-10T19:10:28.779 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:10:29.495 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:10:29.495 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:10:29.328538Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:10:29.328538Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:10:29.328538Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:10:29.328538Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:10:29.328538Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:10:29.328538Z", "running": 0, "size": 1}}] 2022-03-10T19:10:39.516 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:10:40.269 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:10:40.270 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:10:40.048945Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:10:40.048945Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:10:40.048945Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:10:40.048945Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:10:40.048945Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:10:40.048945Z", "running": 0, "size": 1}}] 2022-03-10T19:10:50.298 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:10:51.015 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:10:51.015 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:10:50.854206Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:10:50.854206Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:10:50.854206Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:10:50.854206Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:10:50.854206Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:10:50.854206Z", "running": 0, "size": 1}}] 2022-03-10T19:11:01.039 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:11:01.735 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:11:01.736 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:11:01.530250Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:11:01.530250Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:11:01.530250Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:11:01.530250Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:11:01.530250Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:11:01.530250Z", "running": 0, "size": 1}}] 2022-03-10T19:11:11.759 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:11:12.466 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:11:12.467 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:11:12.280626Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:11:12.280626Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:11:12.280626Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:11:12.280626Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:11:12.280626Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:11:12.280626Z", "running": 0, "size": 1}}] 2022-03-10T19:11:22.488 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:11:23.309 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:11:23.310 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:11:23.030112Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:11:23.030112Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:11:23.030112Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:11:23.030112Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:11:23.030112Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:11:23.030112Z", "running": 0, "size": 1}}] 2022-03-10T19:11:33.332 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:11:34.036 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:11:34.037 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:11:33.876313Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:11:33.876313Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:11:33.876313Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:11:33.876313Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:11:33.876313Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:11:33.876313Z", "running": 0, "size": 1}}] 2022-03-10T19:11:44.061 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:11:44.763 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:11:44.763 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:11:44.586080Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:11:44.586080Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:11:44.586080Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:11:44.586080Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:11:44.586080Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:11:44.586080Z", "running": 0, "size": 1}}] 2022-03-10T19:11:54.786 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:11:55.815 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:11:55.815 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:11:55.319047Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:11:55.319047Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:11:55.319047Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:11:55.319047Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:11:55.319047Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:11:55.319047Z", "running": 0, "size": 1}}] 2022-03-10T19:12:05.839 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:12:06.585 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:12:06.586 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:12:06.389719Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:12:06.389719Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:12:06.389719Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:12:06.389719Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:12:06.389719Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:12:06.389719Z", "running": 0, "size": 1}}] 2022-03-10T19:12:16.606 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:12:17.293 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:12:17.294 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:12:17.124259Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:12:17.124259Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:12:17.124259Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:12:17.124259Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:12:17.124259Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:12:17.124259Z", "running": 0, "size": 1}}] 2022-03-10T19:12:27.317 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:12:28.013 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:12:28.014 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:12:27.849744Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:12:27.849744Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:12:27.849744Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:12:27.849744Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:12:27.849744Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:12:27.849744Z", "running": 0, "size": 1}}] 2022-03-10T19:12:38.034 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:12:38.766 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:12:38.766 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:12:38.566953Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:12:38.566953Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:12:38.566953Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:12:38.566953Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:12:38.566953Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:12:38.566953Z", "running": 0, "size": 1}}] 2022-03-10T19:12:48.791 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:12:49.517 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:12:49.542 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:12:49.279442Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:12:49.279442Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:12:49.279442Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:12:49.279442Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:12:49.279442Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:12:49.279442Z", "running": 0, "size": 1}}] 2022-03-10T19:12:59.544 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:13:00.243 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:13:00.243 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:13:00.016326Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:13:00.016326Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:13:00.016326Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:13:00.016326Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:13:00.016326Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:13:00.016326Z", "running": 0, "size": 1}}] 2022-03-10T19:13:10.266 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:13:10.959 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:13:10.959 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:13:10.791056Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:13:10.791056Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:13:10.791056Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:13:10.791056Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:13:10.791056Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:13:10.791056Z", "running": 0, "size": 1}}] 2022-03-10T19:13:20.982 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:13:21.670 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:13:21.671 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:13:21.490063Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:13:21.490063Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:13:21.490063Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:13:21.490063Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:13:21.490063Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:13:21.490063Z", "running": 0, "size": 1}}] 2022-03-10T19:13:31.694 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:13:32.470 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:13:32.471 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:13:32.302275Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:13:32.302275Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:13:32.302275Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:13:32.302275Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:13:32.302275Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:13:32.302275Z", "running": 0, "size": 1}}] 2022-03-10T19:13:42.492 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:13:43.009021Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:13:43.009021Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:13:43.009021Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:13:43.009021Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:13:43.009021Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:13:43.009021Z", "running": 0, "size": 1}}] 2022-03-10T19:13:53.185 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:13:53.922 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:13:53.923 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:13:53.712771Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:13:53.712771Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:13:53.712771Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:13:53.712771Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:13:53.712771Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:13:53.712771Z", "running": 0, "size": 1}}] 2022-03-10T19:14:03.954 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:14:04.734 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:14:04.735 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:14:04.519944Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:14:04.519944Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:14:04.519944Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:14:04.519944Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:14:04.519944Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:14:04.519944Z", "running": 0, "size": 1}}] 2022-03-10T19:14:14.762 DEBUG:teuthology.orchestra.run.smithi093:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hbwqv -- ceph orch ls -f json 2022-03-10T19:14:15.430 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-03-10T19:14:15.430 INFO:teuthology.orchestra.run.smithi093.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:29.000000Z", "last_refresh": "2022-03-10T19:14:15.267400Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:53:04.000000Z", "last_refresh": "2022-03-10T19:14:15.267400Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:50:28.000000Z", "last_refresh": "2022-03-10T19:14:15.267400Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2d8b0d650431a1b0f93d1633e95eabe6487df85ce3368006a175607d49691b4c", "container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "created": "2022-03-10T18:54:26.000000Z", "last_refresh": "2022-03-10T19:14:15.267400Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T19:14:15.267400Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:78e1eb94d97cd01be975676a7cefdde0c403b210", "last_refresh": "2022-03-10T19:14:15.267400Z", "running": 0, "size": 1}}] 2022-03-10T19:14:15.468 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T19:14:15.469 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T19:14:15.469 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-10T19:14:15.470 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T19:14:15.481 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T19:14:15.496 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T19:14:15.515 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T19:14:15.516 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T19:14:15.536 DEBUG:teuthology.orchestra.remote:smithi093:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T19:14:15.551 DEBUG:teuthology.orchestra.run.smithi093:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-10T19:14:15.613 INFO:teuthology.orchestra.run.smithi093.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-10T19:14:15.626 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_78e1eb94d97cd01be975676a7cefdde0c403b210/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T19:14:15.627 DEBUG:teuthology.orchestra.run.smithi093:> kubectl delete -f cluster.yaml 2022-03-10T19:14:15.715 INFO:teuthology.orchestra.run.smithi093.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-10T19:14:15.720 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:15.717227 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-10T19:14:15.723 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:15.718843 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T19:14:15.724 INFO:tasks.rook.operator.smithi093.stdout:I0310 19:14:15.718995 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T19:14:15.724 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:15.719290 I | operator: successfully started the controller-runtime manager 2022-03-10T19:14:15.745 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:15.742234 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T19:14:15.746 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:15.742259 I | operator: watching all namespaces for Ceph CRs 2022-03-10T19:14:15.746 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:15.742291 I | operator: setting up schemes 2022-03-10T19:14:15.750 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:15.746110 I | operator: setting up the controller-runtime manager 2022-03-10T19:14:16.054 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.050822 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T19:14:16.059 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.053879 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T19:14:16.060 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.053910 I | ceph-cluster-controller: successfully started 2022-03-10T19:14:16.060 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.053981 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T19:14:16.060 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054009 I | ceph-crashcollector-controller: successfully started 2022-03-10T19:14:16.060 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054027 I | ceph-block-pool-controller: successfully started 2022-03-10T19:14:16.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054042 I | ceph-object-store-user-controller: successfully started 2022-03-10T19:14:16.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054066 I | ceph-object-realm-controller: successfully started 2022-03-10T19:14:16.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054087 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T19:14:16.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054111 I | ceph-object-zone-controller: successfully started 2022-03-10T19:14:16.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054226 I | ceph-object-controller: successfully started 2022-03-10T19:14:16.062 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054252 I | ceph-file-controller: successfully started 2022-03-10T19:14:16.062 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054275 I | ceph-nfs-controller: successfully started 2022-03-10T19:14:16.062 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054293 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T19:14:16.062 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054313 I | ceph-client-controller: successfully started 2022-03-10T19:14:16.063 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054336 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T19:14:16.063 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054354 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T19:14:16.063 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054366 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T19:14:16.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054377 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T19:14:16.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054391 I | ceph-bucket-topic: successfully started 2022-03-10T19:14:16.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054405 I | ceph-bucket-notification: successfully started 2022-03-10T19:14:16.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054416 I | ceph-bucket-notification: successfully started 2022-03-10T19:14:16.065 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.054428 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T19:14:16.065 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.055415 I | operator: starting the controller-runtime manager 2022-03-10T19:14:16.166 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.162501 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T19:14:16.167 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.162554 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T19:14:16.167 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.162570 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T19:14:16.170 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.165459 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T19:14:16.176 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.167807 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:16.237 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.233754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:14:16.291 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.288488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:14:16.878 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.874502 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:16.861+0000 7f9e617bb380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:16.878 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:16.861+0000 7f9e617bb380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:16.879 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:16.861+0000 7f9e617bb380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:16.879 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:16.916 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:16.912611 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:17.546 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:17.542179 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:17.529+0000 7fe202aa9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:17.546 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:17.529+0000 7fe202aa9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:17.546 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:17.529+0000 7fe202aa9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:17.547 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:17.577 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:17.571613 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:18.268 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:18.265137 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:18.253+0000 7f3b2df44380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:18.269 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:18.253+0000 7f3b2df44380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:18.269 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:18.253+0000 7f3b2df44380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:18.269 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:18.294 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:18.290869 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:18.987 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:18.983178 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:18.969+0000 7f87178d4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:18.987 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:18.969+0000 7f87178d4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:18.987 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:18.969+0000 7f87178d4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:18.988 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:19.033 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:19.029952 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:19.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:19.665818 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:19.653+0000 7fd4db354380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:19.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:19.653+0000 7fd4db354380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:19.670 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:19.653+0000 7fd4db354380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:19.670 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:19.787 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:19.784081 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:20.507 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:20.503948 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:20.493+0000 7fa95566a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:20.508 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:20.493+0000 7fa95566a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:20.508 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:20.493+0000 7fa95566a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:20.508 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:20.673 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:20.669917 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:21.376 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:21.372230 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:21.361+0000 7f8df4ebb380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:21.376 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:21.361+0000 7f8df4ebb380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:21.376 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:21.361+0000 7f8df4ebb380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:21.376 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:21.705 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:21.701997 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:22.460 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:22.456192 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:22.445+0000 7f719a88b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:22.460 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:22.445+0000 7f719a88b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:22.460 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:22.445+0000 7f719a88b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:22.461 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:23.110 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:23.106757 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:23.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:23.808976 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:23.801+0000 7f1593977380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:23.813 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:23.801+0000 7f1593977380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:23.813 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:23.801+0000 7f1593977380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:23.813 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:25.101 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:25.097572 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:25.810 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:25.806609 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:25.797+0000 7f8cb798e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:25.810 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:25.797+0000 7f8cb798e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:25.810 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:25.797+0000 7f8cb798e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:25.811 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:26.359 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:26.356275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:14:26.371 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:26.368628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:14:28.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:28.374020 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:29.073 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:29.069564 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:29.061+0000 7fd448f54380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:29.074 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:29.061+0000 7fd448f54380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:29.074 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:29.061+0000 7fd448f54380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:29.074 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:34.202 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:34.198830 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:34.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:34.869349 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:34.861+0000 7f5d5514c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:34.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:34.861+0000 7f5d5514c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:34.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:34.861+0000 7f5d5514c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:34.873 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:36.451 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:36.447800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:14:36.463 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:36.460713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:14:45.122 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:45.119304 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:14:45.802 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:45.798921 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:14:45.789+0000 7f5a9be5e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:14:45.803 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:45.789+0000 7f5a9be5e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:14:45.803 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:14:45.789+0000 7f5a9be5e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:14:45.803 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:14:46.562 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:46.558954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:14:46.571 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:46.568722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:14:56.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:56.684723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:14:56.697 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:14:56.694309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:06.291 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:06.288253 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:15:06.784 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:06.781383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:06.887 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:06.884119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:07.059 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:07.055966 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:15:07.044+0000 7fc74870b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:15:07.059 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:15:07.044+0000 7fc74870b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:15:07.060 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:15:07.044+0000 7fc74870b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:15:07.060 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:15:17.036 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:17.032663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:17.048 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:17.045254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:27.125 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:27.121479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:27.137 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:27.134042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:37.235 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:37.232334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:37.248 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:37.245775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:47.360 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:47.356635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:47.371 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:47.368350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:48.029 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:48.025439 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:15:48.652 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:48.648810 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:15:48.639+0000 7f0f85ee8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:15:48.652 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:15:48.639+0000 7f0f85ee8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:15:48.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:15:48.639+0000 7f0f85ee8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:15:48.653 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:15:57.477 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:57.474325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:15:57.490 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:15:57.486846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:07.605 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:07.601787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:07.620 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:07.616861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:17.683 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:17.679713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:17.764 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:17.761068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:27.842 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:27.839218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:27.852 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:27.849263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:37.970 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:37.967301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:37.984 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:37.977869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:48.077 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:48.073780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:48.089 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:48.086005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:58.187 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:58.184008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:16:58.201 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:16:58.198185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:08.299 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:08.296073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:08.311 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:08.308239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:10.582 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:10.578448 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:17:11.222 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:11.218516 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:17:11.206+0000 7f24ecde9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:17:11.222 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:17:11.206+0000 7f24ecde9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:17:11.222 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:17:11.206+0000 7f24ecde9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:17:11.223 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:17:18.418 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:18.415495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:18.431 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:18.427822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:28.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:28.540515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:28.557 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:28.554857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:38.656 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:38.652563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:38.667 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:38.664498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:48.792 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:48.788666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:48.805 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:48.802501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:58.921 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:58.917877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:17:58.971 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:17:58.968128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:09.126 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:09.123492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:09.144 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:09.140586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:19.259 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:19.256031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:19.311 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:19.269125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:29.393 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:29.389624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:29.405 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:29.402064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:39.527 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:39.523386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:39.539 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:39.536067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:49.629 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:49.625762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:49.644 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:49.640662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:59.740 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:59.737196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:18:59.754 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:18:59.750825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:09.827 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:09.823380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:09.838 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:09.835291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:19.964 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:19.959682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:19.996 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:19.990429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:30.109 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:30.105934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:30.121 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:30.118304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:40.275 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:40.271907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:40.288 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:40.284945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:50.380 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:50.375143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:50.393 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:50.389959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:19:55.072 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:55.068887 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:19:55.763 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:19:55.759229 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:19:55.746+0000 7f6cc4f56380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:19:55.764 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:19:55.746+0000 7f6cc4f56380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:19:55.764 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:19:55.746+0000 7f6cc4f56380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:19:55.764 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:20:00.476 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:00.472986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:00.489 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:00.485874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:10.582 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:10.579165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:10.595 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:10.592410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:20.683 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:20.680108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:20.700 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:20.697339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:30.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:30.809556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:30.826 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:30.823316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:40.946 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:40.942536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:40.954 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:40.951788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:51.069 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:51.066228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:20:51.082 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:20:51.079500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:01.209 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:01.206425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:01.221 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:01.218906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:11.299 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:11.296763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:11.312 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:11.309155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:21.380 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:21.377251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:21.391 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:21.388268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:31.565 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:31.561851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:31.581 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:31.578724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:41.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:41.684143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:41.701 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:41.698233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:51.828 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:51.824403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:21:51.839 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:21:51.835864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:02.000 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:01.997208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:02.015 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:02.011987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:12.117 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:12.114444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:12.129 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:12.126391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:22.196 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:22.192686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:22.209 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:22.205104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:32.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:32.268657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:32.284 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:32.282094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:42.347 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:42.344624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:42.360 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:42.357980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:52.430 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:52.427176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:22:52.444 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:22:52.439867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:02.592 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:02.589775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:02.604 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:02.601520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:12.733 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:12.730454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:12.745 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:12.742916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:22.837 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:22.834171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:22.885 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:22.883000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:33.002 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:32.999254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:33.016 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:33.011656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:43.118 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:43.114990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:43.130 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:43.127704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:53.276 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:53.273397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:23:53.287 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:23:53.284596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:03.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:03.381764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:03.396 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:03.394006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:13.458 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:13.455735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:13.514 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:13.511474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:23.577 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:23.574985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:23.621 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:23.618204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:33.728 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:33.725445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:33.739 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:33.737358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:43.833 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:43.830922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:43.847 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:43.844836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:53.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:53.988776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:24:54.007 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:24:54.002441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:04.124 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:04.121896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:04.139 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:04.136913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:14.250 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:14.247664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:14.262 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:14.259239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:23.449 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:23.447159 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:25:24.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:24.089211 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:25:24.080+0000 7fa096160380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:25:24.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:25:24.080+0000 7fa096160380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:25:24.093 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:25:24.080+0000 7fa096160380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:25:24.093 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:25:24.356 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:24.354218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:24.368 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:24.366079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:34.458 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:34.455569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:34.471 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:34.469224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:44.527 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:44.524615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:44.597 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:44.595135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:54.695 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:54.692079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:25:54.708 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:25:54.706014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:04.779 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:04.777207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:04.791 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:04.789302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:14.861 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:14.858495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:14.875 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:14.873008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:24.979 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:24.976916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:24.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:24.990324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:35.057 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:35.054437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:35.068 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:35.066149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:45.198 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:45.195686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:45.211 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:45.208831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:55.316 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:55.313344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:26:55.328 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:26:55.326179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:05.443 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:05.440643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:05.456 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:05.453617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:15.526 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:15.524122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:15.571 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:15.569121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:25.661 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:25.658343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:25.688 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:25.674661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:35.745 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:35.742855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:35.805 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:35.803115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:45.936 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:45.934353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:45.946 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:45.944218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:56.013 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:56.010348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:27:56.025 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:27:56.023212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:06.160 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:06.157924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:06.172 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:06.169773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:16.273 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:16.270714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:16.287 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:16.283716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:26.354 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:26.351720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:26.418 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:26.415611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:36.491 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:36.488292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:36.571 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:36.569052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:46.658 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:46.656308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:46.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:46.684795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:56.808 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:56.806190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:28:56.822 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:28:56.820468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:06.952 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:06.949662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:06.964 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:06.962122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:17.057 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:17.054562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:17.075 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:17.073057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:27.183 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:27.181231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:27.195 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:27.192690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:37.298 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:37.296202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:37.311 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:37.308411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:47.364 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:47.361909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:47.375 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:47.373381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:57.465 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:57.462895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:29:57.478 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:29:57.475098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:07.545 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:07.542203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:07.556 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:07.554429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:17.612 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:17.609595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:17.624 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:17.622160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:27.702 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:27.699486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:27.713 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:27.711564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:37.791 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:37.788359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:37.847 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:37.844665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:47.983 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:47.981203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:47.997 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:47.993788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:58.070 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:58.067601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:30:58.116 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:30:58.114058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:08.217 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:08.214939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:08.234 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:08.230103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:18.336 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:18.333801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:18.347 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:18.344925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:28.431 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:28.428466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:28.442 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:28.439591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:38.509 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:38.506461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:38.522 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:38.519671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:48.592 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:48.589976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:48.604 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:48.601976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:58.723 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:58.719844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:31:58.732 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:31:58.729710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:08.798 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:08.795724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:09.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:08.808285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:18.953 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:18.950932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:18.963 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:18.961169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:29.026 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:29.024197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:29.042 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:29.040586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:39.110 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:39.107822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:39.170 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:39.167844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:49.316 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:49.314010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:49.329 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:49.326593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:59.426 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:59.423870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:32:59.438 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:32:59.436139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:09.503 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:09.500958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:09.515 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:09.513043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:19.650 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:19.647271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:19.662 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:19.660348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:29.760 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:29.757512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:29.773 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:29.770358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:39.864 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:39.861010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:39.876 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:39.873604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:49.937 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:49.934866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:33:49.952 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:33:49.950090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:00.054 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:00.051447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:00.065 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:00.062844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:10.140 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:10.137356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:10.152 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:10.150091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:20.212 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:20.209406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:20.224 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:20.221385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:30.379 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:30.376351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:30.392 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:30.389821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:40.473 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:40.470613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:40.487 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:40.484047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:50.581 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:50.578965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:34:50.594 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:34:50.591883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:00.723 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:00.720783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:00.744 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:00.741568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:10.894 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:10.890423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:10.905 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:10.902643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:20.967 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:20.964846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:20.979 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:20.976847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:31.080 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:31.078147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:31.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:31.089723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:41.199 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:41.196476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:41.210 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:41.208084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:51.266 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:51.263650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:35:51.278 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:35:51.275720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:01.436 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:01.434108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:01.448 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:01.445711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:11.527 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:11.524740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:11.537 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:11.535338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:19.460 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:19.457218 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:36:20.103 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:20.100678 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:36:20.089+0000 7f7cf3059380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:36:20.104 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:36:20.089+0000 7f7cf3059380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:36:20.104 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:36:20.089+0000 7f7cf3059380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:36:20.104 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:36:21.602 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:21.599797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:21.612 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:21.609706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:31.766 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:31.763439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:31.776 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:31.773650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:41.841 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:41.838544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:41.852 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:41.850209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:51.960 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:51.957324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:36:51.978 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:36:51.970543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:02.085 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:02.082572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:02.108 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:02.105981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:12.216 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:12.213496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:12.270 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:12.225573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:22.326 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:22.323476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:22.340 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:22.337474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:32.447 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:32.443996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:32.460 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:32.457470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:42.573 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:42.570169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:42.585 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:42.582685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:52.680 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:52.677069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:37:52.696 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:37:52.693249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:02.806 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:02.803733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:02.820 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:02.817674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:12.879 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:12.876906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:12.893 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:12.890771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:22.995 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:22.993152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:23.063 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:23.060387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:33.126 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:33.123882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:33.138 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:33.136130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:43.288 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:43.285629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:43.302 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:43.299490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:53.373 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:53.370381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:38:53.385 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:38:53.382888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:03.443 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:03.437052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:03.451 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:03.449239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:13.564 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:13.561235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:13.575 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:13.572460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:23.670 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:23.667140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:23.681 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:23.679096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:33.763 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:33.761024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:33.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:33.772688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:43.905 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:43.902456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:43.918 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:43.915257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:54.063 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:54.060911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:39:54.074 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:39:54.071710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:04.182 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:04.179993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:04.194 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:04.190552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:14.283 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:14.279996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:14.293 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:14.290548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:24.425 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:24.422200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:24.443 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:24.437002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:34.502 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:34.499774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:34.512 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:34.509589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:44.664 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:44.661699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:44.676 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:44.673383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:54.743 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:54.740886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:40:54.754 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:40:54.751498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:04.836 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:04.832978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:04.885 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:04.882369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:15.038 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:15.035442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:15.050 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:15.047110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:25.139 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:25.136500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:25.154 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:25.151187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:35.215 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:35.212262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:35.227 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:35.224237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:45.293 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:45.290605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:45.328 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:45.325491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:55.451 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:55.446251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:41:55.459 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:41:55.456104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:05.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:05.530720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:05.545 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:05.543147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:15.606 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:15.602721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:15.712 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:15.709341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:25.775 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:25.772471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:25.785 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:25.782912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:35.853 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:35.850500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:35.864 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:35.862331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:45.971 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:45.968866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:46.021 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:46.018592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:56.154 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:56.151325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:42:56.167 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:42:56.163484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:06.229 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:06.227081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:06.242 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:06.239749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:16.400 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:16.397401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:16.408 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:16.406012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:26.563 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:26.560255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:26.580 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:26.574006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:36.676 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:36.673618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:36.691 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:36.686177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:46.788 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:46.786151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:46.799 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:46.796978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:56.903 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:56.900484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:43:56.917 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:43:56.914514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:07.032 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:07.029293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:07.043 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:07.040397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:17.191 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:17.188304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:17.203 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:17.200624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:27.287 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:27.284983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:27.298 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:27.295810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:37.379 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:37.376476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:37.393 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:37.390323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:47.456 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:47.453551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:47.505 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:47.502802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:57.606 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:57.603753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:44:57.619 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:44:57.616963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:07.685 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:07.682709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:07.697 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:07.694456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:17.763 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:17.761077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:17.776 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:17.774107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:27.849 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:27.846465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:27.860 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:27.857673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:38.003 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:38.001123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:38.014 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:38.012125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:48.162 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:48.159980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:48.174 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:48.171748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:58.290 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:58.288212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:45:58.308 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:45:58.306074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:08.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:08.477270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:08.492 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:08.490101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:18.566 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:18.564016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:18.578 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:18.576187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:28.660 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:28.657993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:28.674 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:28.671274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:38.773 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:38.770600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:38.786 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:38.784547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:48.846 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:48.843335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:48.858 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:48.855224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:59.004 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:59.001522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:46:59.016 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:46:59.014594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:09.114 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:09.111463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:09.126 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:09.124332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:19.281 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:19.279163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:19.296 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:19.294319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:29.395 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:29.392681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:29.409 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:29.407111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:39.470 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:39.467581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:39.481 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:39.479557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:49.565 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:49.562218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:49.610 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:49.607408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:59.691 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:59.688621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:47:59.728 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:47:59.717171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:09.783 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:09.781107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:09.797 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:09.795250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:19.908 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:19.904870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:19.920 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:19.918175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:30.046 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:30.043884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:30.059 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:30.056612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:40.250 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:40.247400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:40.261 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:40.258655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:50.361 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:50.358611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:48:50.374 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:48:50.371794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:00.455 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:00.452831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:00.500 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:00.497633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:10.652 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:10.649959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:10.664 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:10.661556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:20.870 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:20.867535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:20.884 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:20.881405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:31.191 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:31.188748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:31.201 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:31.199587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:41.341 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:41.338623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:41.360 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:41.357586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:51.454 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:51.451444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:49:51.467 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:49:51.465040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:01.528 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:01.526066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:01.540 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:01.537737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:11.646 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:11.643414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:11.659 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:11.656471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:21.763 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:21.760841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:21.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:21.775147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:31.843 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:31.841134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:31.855 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:31.853108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:41.928 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:41.925225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:41.939 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:41.937497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:52.002 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:51.999926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:50:52.014 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:50:52.011999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:02.132 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:02.129666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:02.144 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:02.141507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:12.255 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:12.253007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:12.268 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:12.266387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:22.332 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:22.330120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:22.345 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:22.343104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:32.441 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:32.438300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:32.453 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:32.450761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:42.513 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:42.510905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:42.525 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:42.522774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:52.590 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:52.587770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:51:52.659 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:51:52.656743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:02.753 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:02.750368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:02.793 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:02.791170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:12.902 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:12.899299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:12.912 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:12.909627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:23.011 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:23.008140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:23.027 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:23.020704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:33.102 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:33.098179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:33.114 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:33.111562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:43.185 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:43.181024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:43.198 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:43.195384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:53.264 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:53.261072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:52:53.275 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:52:53.273224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:00.112 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:00.109090 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T19:53:00.766 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:00.763090 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T19:53:00.751+0000 7f265edc1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T19:53:00.767 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:53:00.751+0000 7f265edc1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T19:53:00.768 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T19:53:00.751+0000 7f265edc1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T19:53:00.768 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T19:53:03.380 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:03.377982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:03.392 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:03.390235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:13.527 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:13.524706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:13.540 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:13.538199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:23.600 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:23.597635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:23.612 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:23.610472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:33.743 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:33.740700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:33.980 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:33.752996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:43.823 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:43.820028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:43.834 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:43.832119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:53.921 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:53.918469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:53:53.933 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:53:53.930875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:04.033 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:04.030018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:04.044 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:04.042065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:14.116 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:14.113596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:14.127 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:14.124701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:24.203 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:24.200868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:24.213 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:24.210382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:34.279 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:34.276509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:34.293 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:34.290502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:44.412 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:44.409445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:44.429 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:44.426633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:54.529 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:54.526447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:54:54.541 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:54:54.538646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:04.614 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:04.612010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:04.679 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:04.677097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:14.754 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:14.751364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:14.767 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:14.764403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:24.864 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:24.862004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:24.877 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:24.874512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:34.951 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:34.948623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:34.963 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:34.960794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:45.048 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:45.045589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:45.060 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:45.058057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:55.162 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:55.159526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:55:55.174 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:55:55.171524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:05.271 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:05.268623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:05.284 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:05.280807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:15.370 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:15.368209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:15.383 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:15.381223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:25.437 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:25.435072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:25.450 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:25.447558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:35.512 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:35.509303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:35.525 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:35.522309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:45.613 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:45.610156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:45.625 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:45.622428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:55.719 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:55.717090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:56:55.731 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:56:55.728196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:05.800 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:05.798243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:05.854 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:05.852155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:15.973 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:15.970256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:15.986 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:15.983541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:26.097 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:26.094701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:27.182 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:26.106804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:36.250 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:36.247501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:36.262 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:36.260055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:46.333 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:46.330690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:46.347 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:46.344696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:56.430 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:56.427867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:57:56.441 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:57:56.439256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:06.499 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:06.497010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:06.511 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:06.508461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:16.575 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:16.572861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:16.588 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:16.586087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:26.647 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:26.644706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:26.712 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:26.710201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:36.776 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:36.774068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:36.788 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:36.785732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:46.870 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:46.868286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:46.886 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:46.884170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:56.966 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:56.961241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:58:56.975 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:58:56.972674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:07.048 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:07.046432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:07.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:07.058408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:17.121 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:17.115803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:17.130 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:17.128199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:27.198 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:27.195913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:27.212 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:27.210273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:37.310 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:37.308443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:37.323 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:37.320394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:47.419 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:47.408769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:47.424 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:47.422283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:57.505 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:57.503477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T19:59:57.520 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 19:59:57.518192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:07.609 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:07.606576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:07.624 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:07.621908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:17.752 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:17.750041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:17.764 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:17.762830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:27.876 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:27.873570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:27.886 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:27.884528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:37.959 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:37.956927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:37.973 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:37.970402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:48.047 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:48.044723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:48.113 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:48.111247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:58.259 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:58.257547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:00:58.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:00:58.270583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:08.388 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:08.386467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:08.402 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:08.400725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:18.477 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:18.474876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:18.762 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:18.486818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:28.557 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:28.555304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:28.569 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:28.566962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:38.643 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:38.641572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:38.656 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:38.654649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:48.718 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:48.715900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:48.729 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:48.727479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:58.841 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:58.839596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:01:58.852 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:01:58.850333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:08.976 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:08.974407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:08.989 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:08.987739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:19.095 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:19.092984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:19.107 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:19.105142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:29.218 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:29.216345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:29.230 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:29.228921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:39.339 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:39.336887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:39.352 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:39.350778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:49.439 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:49.437956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:49.454 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:49.452413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:59.538 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:59.536030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:02:59.619 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:02:59.617395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:09.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:09.685043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:09.696 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:09.695082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:19.798 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:19.795769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:19.811 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:19.809776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:29.927 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:29.925233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:29.940 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:29.938593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:40.114 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:40.112789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:40.126 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:40.124454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:50.213 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:50.211566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:03:50.226 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:03:50.224393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:00.304 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:00.302224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:00.315 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:00.313743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:10.435 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:10.433406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:10.447 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:10.446013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:20.542 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:20.540890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:20.554 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:20.552817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:30.615 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:30.614029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:30.689 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:30.687559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:40.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:40.776099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:40.790 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:40.788633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:50.887 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:50.884620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:04:50.899 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:04:50.897980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:01.008 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:01.006761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:01.020 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:01.018500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:11.094 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:11.092322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:11.107 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:11.105470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:21.179 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:21.177973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:21.235 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:21.229410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:31.375 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:31.373446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:31.390 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:31.387712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:41.486 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:41.484802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:41.564 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:41.562905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:51.686 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:51.684862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:05:51.706 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:05:51.701722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:01.775 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:01.773585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:01.786 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:01.785180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:11.845 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:11.843752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:11.857 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:11.855759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:21.923 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:21.921463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:21.935 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:21.933852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:32.010 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:32.008784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:32.021 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:32.019991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:42.136 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:42.134488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:42.146 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:42.145163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:52.208 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:52.206514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:06:52.220 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:06:52.218424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:02.313 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:02.311822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:02.325 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:02.324200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:12.423 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:12.421829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:12.437 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:12.436097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:22.534 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:22.532534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:22.547 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:22.545901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:32.611 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:32.609662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:32.623 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:32.622141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:42.728 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:42.726477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:42.740 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:42.739460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:52.837 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:52.835412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:07:52.848 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:07:52.847246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:02.912 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:02.910449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:02.928 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:02.926536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:13.030 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:13.029150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:13.043 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:13.041838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:23.103 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:23.101485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:23.114 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:23.113479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:33.203 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:33.201558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:33.220 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:33.218798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:43.308 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:43.306416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:43.319 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:43.318250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:53.378 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:53.376791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:08:53.390 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:08:53.389110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:03.489 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:03.487771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:03.502 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:03.500993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:13.578 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:13.576298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:13.590 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:13.588571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:23.681 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:23.679577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:23.708 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:23.706510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:33.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:33.810717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:33.823 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:33.822535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:40.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:40.776322 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T20:09:41.417 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:41.415906 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T20:09:41.403+0000 7fdda2912380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T20:09:41.418 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T20:09:41.403+0000 7fdda2912380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T20:09:41.419 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T20:09:41.403+0000 7fdda2912380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T20:09:41.419 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T20:09:43.950 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:43.948706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:43.965 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:43.963779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:54.093 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:54.091615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:09:54.104 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:09:54.103405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:04.177 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:04.176379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:04.189 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:04.188360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:14.282 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:14.280438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:14.293 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:14.291672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:24.383 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:24.381962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:24.395 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:24.394128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:34.545 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:34.543994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:34.555 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:34.554567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:44.671 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:44.669801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:44.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:44.685810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:54.780 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:54.778959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:10:54.792 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:10:54.791356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:04.896 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:04.894467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:04.908 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:04.906801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:15.007 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:15.005983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:15.017 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:15.016644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:25.129 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:25.128001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:25.141 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:25.140029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:35.207 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:35.205924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:35.220 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:35.219443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:45.363 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:45.361539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:45.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:45.376284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:55.481 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:55.479546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:11:55.493 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:11:55.492213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:05.547 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:05.544879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:05.557 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:05.555724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:15.660 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:15.658853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:15.672 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:15.670991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:25.779 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:25.777559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:25.790 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:25.788218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:35.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:35.871772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:35.883 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:35.882230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:45.944 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:45.943262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:46.009 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:46.008271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:56.075 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:56.074193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:12:56.089 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:12:56.087927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:06.150 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:06.149529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:06.162 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:06.160912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:16.231 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:16.229853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:16.244 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:16.242602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:26.337 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:26.336105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:26.347 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:26.346800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:36.397 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:36.396576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:36.408 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:36.408004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:46.530 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:46.529258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:46.540 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:46.539291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:56.628 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:56.627402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:13:56.638 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:13:56.637368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:06.735 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:06.733824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:06.788 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:06.786959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:16.882 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:16.881212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:16.895 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:16.894483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:26.984 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:26.982567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:26.996 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:26.995267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:37.058 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:37.056787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:37.070 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:37.069665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:47.133 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:47.132150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:47.144 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:47.143382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:57.263 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:57.262149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:14:57.296 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:14:57.294951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:07.427 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:07.426292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:07.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:07.439634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:17.501 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:17.499983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:17.512 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:17.511771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:27.610 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:27.609360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:27.623 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:27.622250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:37.732 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:37.731227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:37.745 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:37.744551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:47.814 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:47.813033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:47.828 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:47.827293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:57.935 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:57.933722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:15:57.944 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:15:57.944001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:08.011 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:08.010188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:08.027 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:08.026164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:18.151 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:18.149901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:18.160 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:18.159560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:28.262 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:28.260636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:28.276 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:28.275098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:38.409 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:38.408092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:38.421 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:38.420549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:48.538 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:48.537077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:48.549 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:48.548744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:58.647 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:58.646422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:16:59.035 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:16:58.658576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:08.799 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:08.797659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:08.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:08.811023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:18.922 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:18.921269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:18.936 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:18.935302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:29.035 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:29.033930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:29.049 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:29.047720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:39.115 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:39.114181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:39.126 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:39.125573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:49.197 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:49.196030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:49.211 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:49.209676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:59.281 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:59.279966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:17:59.294 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:17:59.293469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:09.403 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:09.401924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:09.417 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:09.415811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:19.521 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:19.520497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:19.534 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:19.533646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:29.638 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:29.636583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:29.649 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:29.648831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:39.762 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:39.754549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:39.771 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:39.769844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:49.846 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:49.844702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:18:49.897 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:18:49.895969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:00.136 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:00.135411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:00.150 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:00.148982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:10.235 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:10.234041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:10.251 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:10.249737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:20.360 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:20.358667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:20.372 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:20.371270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:30.482 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:30.481499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:30.495 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:30.493867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:40.599 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:40.597928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:40.609 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:40.608531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:50.719 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:50.717324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:19:50.730 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:19:50.729157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:00.802 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:00.801370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:00.817 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:00.816283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:10.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:10.872611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:10.887 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:10.885203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:20.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:20.959719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:20.973 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:20.971930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:31.048 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:31.047096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:31.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:31.060433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:41.183 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:41.181051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:41.194 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:41.193128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:51.273 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:51.270498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:20:51.284 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:20:51.283080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:01.359 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:01.357598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:01.371 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:01.369298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:11.482 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:11.481415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:11.495 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:11.494775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:21.564 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:21.563233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:21.615 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:21.614177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:31.730 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:31.728836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:31.739 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:31.738545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:41.885 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:41.883628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:41.896 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:41.894863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:51.954 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:51.953137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:21:51.965 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:21:51.964448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:02.068 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:02.067512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:02.081 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:02.080022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:12.151 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:12.149511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:12.163 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:12.162078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:22.262 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:22.260894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:22.300 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:22.299144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:32.359 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:32.357920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:32.372 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:32.371627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:42.438 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:42.437444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:42.449 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:42.448489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:52.543 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:52.541903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:22:52.555 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:22:52.554271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:02.656 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:02.655455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:02.666 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:02.665872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:12.765 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:12.764087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:12.784 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:12.774525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:22.880 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:22.879093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:22.891 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:22.890227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:32.957 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:32.956011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:32.972 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:32.970853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:43.077 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:43.075296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:43.123 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:43.119145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:53.219 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:53.217986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:23:53.230 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:23:53.229950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:03.299 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:03.298370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:03.361 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:03.360388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:13.470 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:13.469105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:13.481 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:13.480551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:23.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:23.543640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:23.556 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:23.555586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:33.621 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:33.619779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:33.630 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:33.629777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:43.797 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:43.796326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:43.809 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:43.808473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:53.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:53.959960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:24:53.972 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:24:53.971159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:04.085 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:04.084482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:04.098 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:04.097307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:14.191 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:14.189788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:14.205 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:14.204027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:24.272 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:24.270763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:24.283 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:24.282529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:34.380 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:34.379665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:34.391 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:34.390635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:44.457 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:44.455759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:44.467 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:44.466205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:54.572 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:54.570919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:25:54.585 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:25:54.584963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:04.655 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:04.654152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:04.670 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:04.669285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:14.798 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:14.797256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:14.807 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:14.806498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:21.448 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:21.447183 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T20:26:22.119 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:22.117782 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T20:26:22.106+0000 7f1e521d1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T20:26:22.119 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T20:26:22.106+0000 7f1e521d1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T20:26:22.120 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T20:26:22.106+0000 7f1e521d1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T20:26:22.120 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T20:26:24.930 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:24.929312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:24.942 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:24.941711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:35.066 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:35.064460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:35.078 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:35.076713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:45.157 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:45.155787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:45.169 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:45.168301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:55.234 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:55.233313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:26:55.247 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:26:55.246349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:05.350 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:05.349087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:05.362 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:05.361264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:15.427 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:15.426692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:15.438 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:15.437187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:25.511 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:25.509804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:25.522 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:25.521176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:35.642 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:35.641480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:35.654 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:35.653124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:45.714 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:45.710002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:45.770 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:45.769048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:55.830 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:55.829099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:27:55.840 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:27:55.839958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:05.995 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:05.994727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:06.007 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:06.006983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:16.121 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:16.120273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:16.163 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:16.159102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:26.234 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:26.233068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:26.291 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:26.290394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:36.395 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:36.394582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:36.461 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:36.460323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:46.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:46.542619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:46.604 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:46.603342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:56.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:56.668802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:28:56.681 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:28:56.680109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:06.747 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:06.746548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:06.810 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:06.809383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:16.885 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:16.883076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:16.895 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:16.893915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:26.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:26.989786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:27.003 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:27.001833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:37.061 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:37.060020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:37.142 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:37.138268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:47.203 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:47.201562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:47.214 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:47.213155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:57.341 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:57.340043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:29:57.354 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:29:57.353190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:07.473 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:07.472057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:07.486 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:07.485502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:17.585 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:17.584384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:17.598 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:17.597430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:27.698 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:27.697329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:27.708 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:27.707133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:37.783 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:37.782509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:37.823 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:37.822583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:47.888 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:47.886807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:47.933 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:47.932881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:57.993 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:57.992696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:30:58.007 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:30:58.004836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:08.065 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:08.064836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:08.077 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:08.077046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:18.181 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:18.180675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:18.194 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:18.193967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:28.259 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:28.258281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:28.270 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:28.269042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:38.345 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:38.344316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:38.361 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:38.360423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:48.458 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:48.456897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:48.472 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:48.471964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:58.588 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:58.587231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:31:58.602 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:31:58.600838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:08.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:08.685634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:08.698 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:08.697629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:18.813 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:18.811632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:18.825 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:18.824297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:28.919 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:28.918067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:28.960 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:28.959136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:39.040 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:39.039610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:39.093 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:39.092207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:49.188 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:49.186746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:49.200 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:49.199142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:59.327 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:59.326191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:32:59.339 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:32:59.338931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:09.408 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:09.407258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:09.421 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:09.420347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:19.537 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:19.535670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:19.554 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:19.553435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:29.678 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:29.676683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:29.690 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:29.688679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:39.789 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:39.787721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:39.803 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:39.802247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:49.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:49.871756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:33:49.953 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:33:49.951896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:00.080 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:00.079198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:00.094 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:00.093755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:10.173 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:10.171592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:10.214 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:10.212791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:20.336 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:20.335171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:20.350 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:20.349108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:30.408 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:30.407525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:30.417 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:30.416807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:40.468 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:40.467536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:40.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:40.479506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:50.580 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:50.578612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:34:50.592 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:34:50.591123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:00.659 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:00.657558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:00.671 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:00.670192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:10.759 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:10.758405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:10.774 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:10.773234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:20.892 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:20.890716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:20.903 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:20.902850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:30.969 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:30.968564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:30.981 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:30.980451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:41.103 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:41.101603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:41.112 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:41.111521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:51.198 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:51.196603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:35:51.209 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:35:51.207896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:01.321 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:01.320117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:01.333 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:01.332557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:11.402 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:11.400716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:11.411 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:11.410105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:21.485 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:21.484471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:21.556 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:21.555323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:31.679 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:31.677637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:31.690 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:31.689129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:41.781 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:41.780238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:41.794 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:41.793204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:51.911 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:51.910264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:36:52.010 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:36:52.008849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:02.107 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:02.102498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:02.116 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:02.114739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:12.237 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:12.235422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:12.250 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:12.248968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:22.317 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:22.315787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:22.328 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:22.327529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:32.391 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:32.390121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:32.404 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:32.402986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:42.510 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:42.509670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:42.522 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:42.521470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:52.592 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:52.591059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:37:52.603 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:37:52.602285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:02.705 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:02.704462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:02.718 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:02.717170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:12.801 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:12.800048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:12.811 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:12.810611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:22.875 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:22.874797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:22.925 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:22.922798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:33.011 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:33.010463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:33.022 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:33.021342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:43.140 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:43.139391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:43.154 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:43.153374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:53.266 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:53.264639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:38:53.279 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:38:53.277982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:03.347 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:03.346534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:03.359 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:03.358313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:13.458 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:13.456384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:13.468 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:13.467342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:23.534 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:23.532908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:23.549 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:23.547389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:33.693 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:33.692025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:33.707 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:33.706379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:43.796 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:43.795609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:43.820 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:43.819420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:53.944 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:53.942867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:39:53.955 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:39:53.954609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:04.052 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:04.051193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:04.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:04.063710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:14.162 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:14.158973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:14.171 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:14.170751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:24.238 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:24.234494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:24.248 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:24.247050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:34.304 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:34.303594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:34.315 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:34.314124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:44.411 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:44.410440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:44.424 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:44.423032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:54.537 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:54.536938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:40:54.550 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:40:54.549393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:04.622 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:04.621187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:04.631 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:04.630647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:14.735 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:14.734696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:14.792 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:14.790882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:24.895 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:24.894492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:24.906 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:24.905771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:34.972 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:34.971379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:34.985 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:34.984670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:45.079 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:45.077972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:45.091 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:45.090526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:55.149 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:55.148933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:41:55.162 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:41:55.161802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:05.229 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:05.228249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:05.245 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:05.244528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:15.483 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:15.482483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:15.495 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:15.494707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:25.614 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:25.612875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:25.627 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:25.626308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:35.797 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:35.796895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:35.808 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:35.807921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:45.903 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:45.902022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:45.912 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:45.912220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:56.021 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:56.020308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:42:56.033 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:42:56.032371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:02.125 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:02.124876 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T20:43:02.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:02.811228 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T20:43:02.800+0000 7f0807bb4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T20:43:02.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T20:43:02.800+0000 7f0807bb4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T20:43:02.813 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T20:43:02.800+0000 7f0807bb4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T20:43:02.813 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T20:43:06.112 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:06.111144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:06.177 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:06.176370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:16.244 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:16.243168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:16.253 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:16.253004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:26.367 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:26.366703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:26.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:26.375509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:36.451 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:36.450775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:36.466 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:36.465375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:46.567 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:46.566263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:46.578 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:46.578173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:56.652 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:56.651889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:43:56.667 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:43:56.666453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:06.756 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:06.755490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:06.767 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:06.767086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:16.835 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:16.835277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:16.846 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:16.846295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:26.977 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:26.977106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:26.987 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:26.987155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:37.058 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:37.057577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:37.072 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:37.071594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:47.206 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:47.205446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:47.219 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:47.218698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:57.287 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:57.286672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:44:57.295 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:44:57.295452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:07.352 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:07.351996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:07.365 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:07.364231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:17.437 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:17.436602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:17.449 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:17.448528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:27.526 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:27.525567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:27.541 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:27.540634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:37.662 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:37.661674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:37.675 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:37.674333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:47.778 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:47.777304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:47.816 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:47.815123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:57.945 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:57.944368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:45:57.957 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:45:57.956929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:08.022 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:08.021886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:08.114 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:08.114386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:18.201 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:18.201152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:18.213 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:18.213236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:28.312 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:28.311423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:28.324 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:28.324138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:38.388 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:38.387599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:38.400 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:38.400479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:48.530 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:48.530214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:48.539 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:48.539654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:58.608 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:58.608285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:46:58.621 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:46:58.620716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:08.713 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:08.712507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:08.724 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:08.723710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:18.833 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:18.832689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:18.845 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:18.844981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:28.970 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:28.969525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:28.982 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:28.982252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:39.032 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:39.031954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:39.044 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:39.044273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:49.148 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:49.148227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:49.161 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:49.161273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:59.259 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:59.258925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:47:59.382 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:47:59.271117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:09.368 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:09.367987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:09.385 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:09.384678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:19.453 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:19.452763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:19.465 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:19.464360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:29.546 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:29.545337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:29.558 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:29.557992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:39.628 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:39.628321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:39.641 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:39.641351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:49.750 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:49.749771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:49.766 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:49.764452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:59.851 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:59.850387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:48:59.863 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:48:59.863344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:09.934 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:09.934011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:09.943 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:09.943530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:20.027 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:20.026500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:20.037 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:20.036482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:30.116 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:30.115836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:30.127 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:30.127314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:40.191 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:40.191158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:40.205 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:40.203976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:50.325 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:50.325079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:49:50.336 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:49:50.336183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:00.417 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:00.416933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:00.474 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:00.474297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:10.561 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:10.561287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:10.574 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:10.574152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:20.643 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:20.643181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:20.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:20.653515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:30.746 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:30.745250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:30.757 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:30.757004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:40.880 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:40.879635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:40.894 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:40.891992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:51.006 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:51.006211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:50:51.018 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:50:51.018433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:01.112 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:01.112213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:01.123 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:01.123361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:11.245 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:11.245142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:11.256 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:11.256285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:21.423 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:21.422630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:21.435 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:21.434953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:31.602 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:31.601870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:31.617 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:31.616250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:41.683 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:41.682994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:41.768 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:41.766054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:51.858 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:51.857980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:51:51.871 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:51:51.871082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:01.935 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:01.935167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:01.978 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:01.977841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:12.085 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:12.084668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:12.096 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:12.096361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:22.165 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:22.163473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:22.175 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:22.175063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:32.253 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:32.252854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:32.266 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:32.266597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:42.392 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:42.391950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:42.402 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:42.402235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:52.501 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:52.501303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:52:52.516 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:52:52.515306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:02.587 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:02.587439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:02.600 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:02.600164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:12.663 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:12.662999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:12.674 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:12.674316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:22.799 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:22.799529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:22.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:22.812310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:32.908 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:32.908075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:32.918 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:32.918163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:43.040 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:43.038802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:43.049 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:43.050063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:53.110 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:53.110069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:53:53.171 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:53:53.171137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:03.267 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:03.266894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:03.283 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:03.279396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:13.419 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:13.419610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:13.429 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:13.429100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:23.534 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:23.533768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:23.547 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:23.547841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:33.615 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:33.614438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:33.676 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:33.676203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:43.739 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:43.739185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:43.750 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:43.750601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:53.820 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:53.820587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:54:53.832 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:54:53.832848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:03.969 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:03.968901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:03.984 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:03.983732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:14.113 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:14.113087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:14.144 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:14.143149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:24.241 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:24.241824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:24.252 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:24.252679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:34.353 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:34.353096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:34.364 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:34.364620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:44.449 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:44.449535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:44.461 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:44.461917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:54.630 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:54.630418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:55:54.643 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:55:54.643727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:04.765 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:04.765082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:04.779 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:04.779489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:14.886 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:14.885887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:14.899 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:14.899287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:25.008 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:25.007875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:25.017 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:25.017674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:35.077 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:35.077142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:35.133 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:35.133287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:45.230 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:45.230201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:45.243 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:45.243698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:55.353 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:55.352995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:56:55.370 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:56:55.369794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:05.469 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:05.468924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:05.482 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:05.482254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:15.546 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:15.546080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:15.559 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:15.559580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:25.671 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:25.671169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:25.683 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:25.683845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:35.765 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:35.765625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:35.781 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:35.777844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:45.836 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:45.836558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:45.893 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:45.893623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:56.007 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:56.006849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:57:56.019 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:57:56.019406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:06.113 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:06.111142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:06.124 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:06.124252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:16.220 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:16.220053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:16.232 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:16.232449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:26.302 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:26.302435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:26.315 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:26.314662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:36.364 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:36.364520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:36.407 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:36.408045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:46.482 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:46.482506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:46.491 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:46.491709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:56.567 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:56.567453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:58:56.578 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:58:56.578714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:06.681 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:06.681587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:06.693 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:06.693628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:16.789 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:16.788950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:16.801 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:16.801442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:26.876 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:26.876823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:26.887 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:26.887195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:37.003 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:37.003716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:37.017 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:37.017583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:42.820 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:42.820349 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T20:59:43.512 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:43.512139 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T20:59:43.498+0000 7fed36930380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T20:59:43.513 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T20:59:43.498+0000 7fed36930380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T20:59:43.513 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T20:59:43.498+0000 7fed36930380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T20:59:43.513 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T20:59:47.099 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:47.099483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:47.112 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:47.112747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:57.278 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:57.278059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T20:59:57.289 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 20:59:57.289538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:07.442 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:07.442023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:07.452 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:07.452380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:17.517 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:17.516820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:17.528 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:17.528405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:27.640 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:27.639929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:27.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:27.653018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:37.759 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:37.758704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:37.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:37.777891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:47.877 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:47.877128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:47.894 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:47.894213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:57.968 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:57.968753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:00:57.986 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:00:57.980861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:08.045 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:08.045672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:08.117 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:08.117150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:18.203 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:18.203496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:18.213 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:18.213859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:28.317 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:28.317060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:28.329 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:28.329983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:38.445 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:38.445292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:38.458 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:38.458367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:48.543 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:48.543119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:48.555 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:48.555075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:58.626 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:58.626247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:01:58.638 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:01:58.638476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:08.737 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:08.737891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:08.751 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:08.751387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:18.855 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:18.855334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:18.867 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:18.868011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:28.929 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:28.929061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:28.939 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:28.939921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:39.016 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:39.016349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:39.030 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:39.030404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:49.088 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:49.088605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:49.100 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:49.100603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:59.181 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:59.180788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:02:59.194 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:02:59.194958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:09.267 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:09.267710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:09.278 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:09.279155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:19.381 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:19.381891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:19.396 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:19.396452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:29.459 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:29.459905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:29.471 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:29.471396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:39.535 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:39.535376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:39.546 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:39.547058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:49.674 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:49.674377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:49.686 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:49.685743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:59.754 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:59.754670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:03:59.764 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:03:59.764357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:09.925 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:09.924985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:09.933 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:09.933885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:20.013 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:20.013738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:20.074 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:20.074603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:30.179 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:30.179140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:30.192 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:30.192948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:40.302 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:40.302471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:40.311 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:40.312318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:50.429 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:50.429007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:04:50.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:04:50.440366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:00.529 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:00.529177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:00.540 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:00.540483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:10.616 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:10.616618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:10.627 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:10.627942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:20.716 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:20.716573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:20.729 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:20.729932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:30.802 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:30.802422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:30.815 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:30.815946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:40.907 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:40.907404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:40.920 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:40.921169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:51.015 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:51.015333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:05:51.023 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:05:51.024366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:01.124 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:01.124772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:01.137 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:01.137734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:11.217 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:11.217554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:11.227 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:11.227487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:21.318 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:21.318333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:21.333 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:21.333232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:31.466 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:31.466768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:31.479 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:31.479126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:41.706 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:41.706102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:41.718 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:41.719035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:51.834 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:51.834885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:06:51.848 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:06:51.848614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:01.955 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:01.955448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:01.967 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:01.967848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:12.072 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:12.072362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:12.084 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:12.085013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:22.150 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:22.150324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:22.210 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:22.210790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:32.312 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:32.312007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:32.324 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:32.325045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:42.419 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:42.418939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:42.433 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:42.432892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:52.499 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:52.499447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:07:52.512 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:07:52.512741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:02.579 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:02.579568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:02.592 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:02.592338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:12.655 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:12.655650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:12.668 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:12.669097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:22.781 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:22.781169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:22.829 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:22.792043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:32.915 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:32.916027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:32.929 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:32.929649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:42.998 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:42.998905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:43.009 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:43.010026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:53.129 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:53.129835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:08:53.143 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:08:53.143894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:03.206 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:03.206740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:03.263 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:03.263285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:13.340 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:13.340648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:13.352 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:13.352715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:23.467 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:23.467665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:23.478 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:23.479062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:33.572 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:33.572280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:33.583 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:33.584316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:43.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:43.653927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:43.713 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:43.713367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:53.781 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:53.781301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:09:53.790 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:09:53.790951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:03.878 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:03.878651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:03.889 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:03.890137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:13.972 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:13.972629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:14.040 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:14.040982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:24.104 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:24.104386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:24.153 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:24.153913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:34.249 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:34.249132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:34.262 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:34.262135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:44.351 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:44.351224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:44.362 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:44.361900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:54.484 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:54.484846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:10:54.494 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:10:54.494806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:04.579 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:04.579276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:04.591 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:04.589778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:14.666 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:14.666559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:14.680 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:14.680817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:24.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:24.777455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:24.790 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:24.790066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:34.901 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:34.900912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:34.913 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:34.913822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:45.019 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:45.019140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:45.030 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:45.030616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:55.144 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:55.144252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:11:55.155 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:11:55.155672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:05.220 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:05.220698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:05.234 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:05.234142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:15.353 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:15.353223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:15.366 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:15.366559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:25.481 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:25.481088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:25.493 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:25.492458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:35.575 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:35.575559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:35.589 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:35.589005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:45.647 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:45.647524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:45.657 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:45.657534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:55.715 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:55.714870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:12:55.727 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:12:55.727711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:05.796 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:05.796316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:05.812 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:05.812190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:15.868 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:15.868114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:15.881 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:15.881617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:25.980 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:25.970784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:25.985 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:25.985076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:36.042 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:36.042253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:36.053 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:36.053551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:46.141 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:46.141224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:46.153 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:46.151922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:56.240 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:56.240682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:13:56.250 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:13:56.250634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:06.322 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:06.321748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:06.332 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:06.331836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:16.422 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:16.422416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:16.433 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:16.433311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:26.498 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:26.497955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:26.628 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:26.628552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:36.723 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:36.723540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:36.734 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:36.734299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:46.802 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:46.802212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:46.813 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:46.813521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:56.878 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:56.877624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:14:56.948 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:14:56.890335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:07.000 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:06.999931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:07.012 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:07.012744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:17.093 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:17.093171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:17.154 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:17.153776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:27.212 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:27.212004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:27.225 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:27.224233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:37.320 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:37.319776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:37.332 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:37.332037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:47.403 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:47.403370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:47.418 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:47.417920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:57.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:57.533064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:15:57.543 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:15:57.543066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:07.625 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:07.624393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:07.699 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:07.699170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:17.830 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:17.829913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:17.844 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:17.844228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:23.533 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:23.532357 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T21:16:24.195 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:24.194352 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T21:16:24.184+0000 7fbbc52d2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T21:16:24.195 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T21:16:24.184+0000 7fbbc52d2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T21:16:24.195 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T21:16:24.184+0000 7fbbc52d2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T21:16:24.196 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T21:16:28.041 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:28.040481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:28.052 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:28.051828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:38.120 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:38.119412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:38.130 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:38.129969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:48.235 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:48.234140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:48.246 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:48.245983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:58.320 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:58.318689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:16:58.382 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:16:58.381699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:08.468 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:08.467213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:08.530 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:08.529743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:18.605 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:18.604877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:18.617 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:18.616831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:28.681 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:28.680125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:28.691 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:28.690702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:38.750 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:38.750109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:38.763 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:38.762321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:48.862 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:48.861839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:48.874 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:48.874034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:58.981 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:58.980749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:17:58.995 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:17:58.994372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:09.115 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:09.114451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:09.127 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:09.126812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:19.250 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:19.250318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:19.265 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:19.264263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:29.375 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:29.374929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:29.387 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:29.386381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:39.470 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:39.470105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:39.482 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:39.482371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:49.545 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:49.544664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:49.563 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:49.563077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:59.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:59.668557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:18:59.681 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:18:59.680284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:09.744 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:09.743876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:09.757 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:09.756492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:19.872 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:19.871608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:19.884 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:19.884325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:29.947 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:29.946796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:29.962 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:29.961427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:40.026 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:40.025852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:40.037 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:40.036850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:50.149 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:50.148342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:19:50.168 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:19:50.168150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:00.238 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:00.237598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:00.253 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:00.253009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:10.312 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:10.311374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:10.321 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:10.320811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:20.462 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:20.461115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:20.483 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:20.483186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:30.601 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:30.600977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:30.613 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:30.612490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:40.680 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:40.679603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:40.747 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:40.746363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:50.885 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:50.884184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:20:50.896 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:20:50.895960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:00.999 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:00.998968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:01.012 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:01.011880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:11.121 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:11.119485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:11.132 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:11.131502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:21.232 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:21.231066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:21.244 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:21.243415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:31.314 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:31.313494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:31.364 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:31.363299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:41.460 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:41.459431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:41.473 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:41.472491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:51.615 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:51.614393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:21:51.646 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:21:51.644910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:01.793 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:01.792012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:01.802 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:01.801513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:11.885 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:11.884599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:11.897 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:11.896910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:21.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:21.990886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:22.003 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:22.001787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:32.101 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:32.100223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:32.115 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:32.113004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:42.192 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:42.191418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:42.203 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:42.202918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:52.269 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:52.268384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:22:52.281 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:22:52.280656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:02.363 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:02.362662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:02.377 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:02.375597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:12.470 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:12.469312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:12.482 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:12.481189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:22.553 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:22.552020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:22.565 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:22.564175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:32.632 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:32.631649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:32.694 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:32.692408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:42.770 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:42.765546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:42.782 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:42.781715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:52.861 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:52.860683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:23:52.921 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:23:52.920565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:02.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:02.991160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:03.003 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:03.002690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:13.133 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:13.132742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:13.146 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:13.145656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:23.270 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:23.269234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:23.281 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:23.280532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:33.435 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:33.434358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:33.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:33.479106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:43.585 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:43.583987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:43.597 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:43.597103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:53.726 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:53.725370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:24:53.735 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:24:53.735139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:03.822 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:03.821227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:03.834 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:03.833305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:13.964 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:13.963522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:13.978 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:13.976094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:24.043 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:24.042150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:24.054 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:24.053542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:34.147 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:34.146113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:34.161 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:34.158950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:44.235 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:44.234567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:44.246 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:44.245599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:54.302 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:54.301665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:25:54.367 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:25:54.366662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:04.470 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:04.469607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:04.482 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:04.481314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:14.550 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:14.549809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:14.562 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:14.561812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:24.684 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:24.682973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:24.703 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:24.702192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:34.810 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:34.809020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:34.822 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:34.821846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:44.927 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:44.926673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:44.939 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:44.938572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:55.007 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:55.006000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:26:55.017 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:26:55.016742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:05.086 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:05.085061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:05.099 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:05.098971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:15.337 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:15.336113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:15.350 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:15.349345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:25.449 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:25.448580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:25.461 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:25.460419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:35.563 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:35.562193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:35.573 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:35.573033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:45.689 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:45.688516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:45.699 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:45.698924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:55.766 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:55.765543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:27:55.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:27:55.776684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:05.844 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:05.843020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:05.855 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:05.855024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:15.935 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:15.934720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:15.948 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:15.947246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:26.017 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:26.016548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:26.028 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:26.027479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:36.096 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:36.095801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:36.108 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:36.107935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:46.175 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:46.174057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:46.186 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:46.185923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:56.277 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:56.276571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:28:56.291 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:28:56.290202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:06.360 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:06.359555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:06.373 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:06.372598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:16.523 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:16.522242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:16.541 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:16.540257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:26.601 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:26.600823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:26.699 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:26.698533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:36.809 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:36.807860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:36.822 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:36.821290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:46.879 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:46.878541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:46.892 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:46.891272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:56.982 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:56.981291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:29:56.996 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:29:56.995082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:07.070 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:07.066155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:07.087 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:07.086259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:17.179 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:17.178746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:17.193 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:17.192656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:27.293 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:27.292111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:27.303 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:27.302297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:37.380 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:37.379032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:37.439 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:37.438487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:47.540 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:47.539312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:47.551 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:47.550862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:57.616 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:57.615817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:30:57.628 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:30:57.627850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:07.742 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:07.741665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:07.756 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:07.755430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:17.866 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:17.864905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:17.878 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:17.877961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:27.993 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:27.992783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:28.006 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:28.005358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:38.122 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:38.121488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:38.134 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:38.133477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:48.196 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:48.195656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:48.212 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:48.211820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:58.315 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:58.314150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:31:58.328 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:31:58.326293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:08.418 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:08.417872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:08.431 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:08.430429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:18.488 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:18.487104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:18.499 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:18.498840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:28.599 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:28.598348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:28.613 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:28.612915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:38.685 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:38.684369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:38.697 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:38.696813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:48.771 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:48.770496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:48.785 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:48.784640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:58.849 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:58.848201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:32:58.861 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:32:58.860962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:04.202 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:04.201475 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T21:33:04.895 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:04.893327 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T21:33:04.883+0000 7f5297451380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T21:33:04.895 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T21:33:04.883+0000 7f5297451380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T21:33:04.896 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T21:33:04.883+0000 7f5297451380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T21:33:04.896 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T21:33:08.949 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:08.948876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:08.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:08.960938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:19.051 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:19.050022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:19.110 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:19.109778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:29.174 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:29.173655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:29.186 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:29.185977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:39.250 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:39.249745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:39.262 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:39.261844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:49.332 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:49.331163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:49.343 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:49.343032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:59.445 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:59.444517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:33:59.593 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:33:59.456345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:09.551 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:09.550131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:09.562 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:09.561864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:19.634 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:19.632835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:19.645 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:19.644099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:29.706 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:29.705949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:29.717 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:29.716877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:39.800 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:39.795990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:39.807 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:39.806585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:49.869 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:49.868395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:49.882 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:49.881342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:59.982 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:59.978319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:34:59.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:34:59.990436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:10.065 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:10.063776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:10.075 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:10.075098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:20.147 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:20.139002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:20.151 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:20.150344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:30.250 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:30.241595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:30.255 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:30.254479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:40.352 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:40.351171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:40.362 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:40.361822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:50.422 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:50.421748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:35:50.436 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:35:50.435520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:00.569 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:00.568350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:00.583 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:00.582861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:10.647 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:10.646195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:10.659 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:10.658299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:20.723 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:20.722149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:20.735 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:20.734482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:30.797 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:30.796313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:30.809 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:30.808292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:40.916 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:40.915075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:40.929 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:40.928277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:51.028 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:51.027399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:36:51.040 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:36:51.039222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:01.136 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:01.135367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:01.148 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:01.147866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:11.274 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:11.273340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:11.290 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:11.289325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:21.416 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:21.415409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:21.432 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:21.431557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:31.518 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:31.517459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:31.565 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:31.564248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:41.620 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:41.619543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:41.631 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:41.630893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:51.771 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:51.770205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:37:51.782 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:37:51.781619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:01.887 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:01.885996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:01.900 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:01.899029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:12.030 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:12.029472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:12.042 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:12.041009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:22.127 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:22.125910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:22.139 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:22.138214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:32.200 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:32.198964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:32.212 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:32.211896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:42.362 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:42.361362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:42.374 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:42.373263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:52.497 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:52.496426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:38:52.510 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:38:52.509230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:02.649 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:02.648327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:02.661 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:02.660662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:12.784 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:12.783246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:12.798 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:12.797132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:22.960 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:22.959449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:22.970 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:22.970038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:33.030 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:33.029060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:33.040 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:33.040249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:43.154 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:43.153141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:43.167 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:43.166559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:53.330 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:53.329212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:39:53.343 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:39:53.342458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:03.497 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:03.496051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:03.509 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:03.508989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:13.641 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:13.640672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:13.654 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:13.653048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:23.734 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:23.727992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:23.741 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:23.740697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:33.819 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:33.818627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:33.833 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:33.831322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:43.939 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:43.938169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:43.993 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:43.992254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:54.056 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:54.055172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:40:54.067 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:40:54.067143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:04.165 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:04.164553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:04.181 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:04.180387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:14.240 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:14.238851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:14.251 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:14.250569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:24.378 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:24.377518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:24.391 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:24.390638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:34.498 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:34.497137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:34.511 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:34.510114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:44.598 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:44.597501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:44.609 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:44.608595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:54.670 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:54.669833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:41:54.722 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:41:54.721804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:04.814 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:04.813861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:04.828 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:04.827430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:14.940 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:14.938944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:14.953 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:14.952856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:25.052 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:25.051598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:25.065 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:25.064410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:35.136 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:35.135113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:35.147 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:35.147042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:45.269 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:45.268369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:45.286 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:45.282450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:55.400 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:55.399487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:42:55.412 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:42:55.412159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:05.509 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:05.508135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:05.521 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:05.521200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:15.588 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:15.587281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:15.601 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:15.600916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:25.673 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:25.672236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:25.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:25.686806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:35.794 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:35.793198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:35.805 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:35.804541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:45.870 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:45.869326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:45.880 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:45.880096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:56.004 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:56.003105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:43:56.015 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:43:56.014784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:06.089 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:06.088543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:06.101 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:06.100297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:16.158 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:16.157199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:16.170 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:16.169049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:26.243 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:26.242125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:26.252 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:26.252099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:36.350 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:36.349552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:36.361 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:36.361206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:46.433 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:46.433008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:46.447 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:46.446808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:56.535 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:56.534976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:44:56.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:44:56.543707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:06.645 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:06.644866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:06.659 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:06.658113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:16.780 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:16.779544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:16.794 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:16.794190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:26.855 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:26.854117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:26.869 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:26.867908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:36.929 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:36.928932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:36.943 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:36.942081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:47.041 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:47.039983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:47.055 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:47.054209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:57.153 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:57.151980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:45:57.165 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:45:57.164359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:07.231 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:07.230022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:07.240 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:07.239947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:17.348 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:17.347569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:17.360 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:17.359977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:27.478 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:27.476983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:27.491 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:27.490909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:37.589 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:37.588417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:37.600 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:37.600042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:47.668 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:47.667577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:47.682 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:47.681588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:57.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:57.775463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:46:57.787 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:46:57.786614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:07.890 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:07.889425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:07.904 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:07.903734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:18.015 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:18.014391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:18.025 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:18.024780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:28.089 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:28.088502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:28.101 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:28.100623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:38.199 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:38.198494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:38.214 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:38.213957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:48.294 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:48.293357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:48.307 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:48.306899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:58.401 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:58.400375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:47:58.413 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:47:58.412518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:08.505 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:08.504303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:08.517 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:08.516980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:18.593 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:18.592893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:18.664 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:18.663421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:28.796 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:28.795757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:28.809 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:28.808800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:38.894 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:38.892525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:39.044 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:38.906086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:49.005 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:49.004758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:49.018 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:49.017219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:59.128 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:59.127163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:48:59.140 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:48:59.139798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:09.206 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:09.205490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:09.219 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:09.218604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:19.278 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:19.277803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:19.293 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:19.293267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:29.407 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:29.405973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:29.419 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:29.418480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:39.479 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:39.478196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:39.491 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:39.490648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:44.901 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:44.900888 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T21:49:45.556 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:45.554998 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T21:49:45.545+0000 7f1ee74f1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T21:49:45.557 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T21:49:45.545+0000 7f1ee74f1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T21:49:45.557 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T21:49:45.545+0000 7f1ee74f1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T21:49:45.557 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T21:49:49.574 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:49.573157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:49.584 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:49.583868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:59.680 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:59.678701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:49:59.692 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:49:59.692152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:09.859 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:09.859093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:09.872 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:09.871885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:19.936 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:19.933078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:19.982 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:19.981860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:30.107 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:30.106575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:30.121 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:30.120642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:40.234 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:40.232455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:40.245 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:40.244983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:50.360 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:50.359681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:50:50.372 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:50:50.371853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:00.465 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:00.464599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:00.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:00.478876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:10.603 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:10.602567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:10.616 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:10.615638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:20.708 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:20.707725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:20.723 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:20.723052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:30.814 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:30.813129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:30.826 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:30.825701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:40.896 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:40.895253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:40.907 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:40.906628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:50.996 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:50.995252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:51:51.008 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:51:51.007589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:01.122 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:01.121196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:01.135 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:01.134802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:11.204 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:11.203925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:11.216 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:11.215710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:21.305 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:21.304586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:21.319 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:21.318879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:31.450 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:31.449843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:31.463 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:31.462745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:41.590 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:41.589767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:41.602 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:41.601798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:51.674 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:51.673816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:52:51.686 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:52:51.685812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:01.795 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:01.794268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:01.807 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:01.806789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:12.058 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:12.057125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:12.072 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:12.071576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:22.240 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:22.238977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:22.254 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:22.253573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:32.423 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:32.422786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:32.435 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:32.435044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:42.502 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:42.501960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:42.516 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:42.515985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:52.575 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:52.574320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:53:52.586 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:53:52.585568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:02.706 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:02.705677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:02.718 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:02.717478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:12.836 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:12.835803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:12.850 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:12.849847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:22.957 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:22.956125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:22.970 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:22.969268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:33.082 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:33.081744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:33.094 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:33.094215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:43.167 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:43.166942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:43.179 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:43.179187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:53.254 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:53.253826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:54:53.268 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:54:53.267356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:03.344 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:03.343149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:03.357 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:03.356016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:13.422 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:13.421211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:13.474 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:13.473765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:23.567 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:23.566420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:23.580 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:23.580253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:33.695 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:33.694177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:33.703 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:33.702787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:43.829 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:43.828413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:43.840 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:43.840239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:53.908 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:53.907211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:55:53.954 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:55:53.953355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:04.032 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:04.031307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:04.044 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:04.043437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:14.163 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:14.162779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:14.174 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:14.173981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:24.286 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:24.285749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:24.297 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:24.296811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:34.423 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:34.422265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:34.438 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:34.436838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:44.522 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:44.512397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:44.526 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:44.525888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:54.605 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:54.604750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:56:54.692 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:56:54.691632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:04.805 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:04.804888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:04.818 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:04.816854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:14.953 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:14.952706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:14.964 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:14.963673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:25.129 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:25.128646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:25.143 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:25.143312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:35.246 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:35.245930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:35.261 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:35.260525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:45.356 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:45.355987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:45.368 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:45.367936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:55.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:55.439510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:57:55.452 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:57:55.451474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:05.540 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:05.539638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:05.554 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:05.552388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:15.654 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:15.653333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:15.666 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:15.666141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:25.760 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:25.759651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:25.772 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:25.772360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:35.838 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:35.837566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:35.852 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:35.851091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:45.908 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:45.907328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:45.920 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:45.919482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:55.985 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:55.985088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:58:55.998 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:58:55.997526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:06.065 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:06.064171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:06.076 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:06.075311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:16.140 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:16.139910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:16.153 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:16.152887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:26.222 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:26.221657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:26.237 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:26.236124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:36.305 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:36.304882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:36.317 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:36.316741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:46.374 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:46.373516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:46.386 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:46.384973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:56.488 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:56.487161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T21:59:56.498 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 21:59:56.497853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:06.559 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:06.558521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:06.570 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:06.570216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:16.628 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:16.627151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:16.638 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:16.637240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:26.694 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:26.694048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:26.705 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:26.704602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:36.813 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:36.812659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:36.828 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:36.827516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:46.933 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:46.932638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:46.944 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:46.943907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:57.043 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:57.043090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:00:57.056 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:00:57.055331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:07.138 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:07.137491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:07.151 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:07.151288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:17.215 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:17.214690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:17.226 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:17.226204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:27.371 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:27.370296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:27.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:27.383203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:37.455 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:37.454857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:37.467 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:37.467165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:47.519 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:47.517728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:47.588 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:47.587714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:57.656 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:57.655880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:01:57.719 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:01:57.718664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:07.802 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:07.801578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:07.814 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:07.813578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:17.928 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:17.927207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:17.939 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:17.938728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:27.998 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:27.997022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:28.010 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:28.009868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:38.079 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:38.078903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:38.129 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:38.128077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:48.232 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:48.231729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:48.243 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:48.243001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:58.312 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:58.311238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:02:58.324 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:02:58.323923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:08.462 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:08.461014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:08.475 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:08.474614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:18.542 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:18.539425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:18.601 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:18.600277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:28.695 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:28.695126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:28.704 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:28.704166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:38.781 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:38.780807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:38.792 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:38.791899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:48.861 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:48.860450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:48.874 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:48.872211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:58.986 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:58.985715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:03:59.000 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:03:58.999948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:09.068 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:09.067366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:09.125 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:09.124770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:19.247 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:19.246612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:19.260 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:19.259921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:29.366 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:29.365578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:29.380 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:29.379428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:39.491 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:39.490165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:39.506 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:39.505359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:49.579 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:49.578946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:49.592 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:49.591943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:59.657 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:59.656286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:04:59.669 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:04:59.668748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:09.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:09.776590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:09.791 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:09.790353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:19.888 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:19.887289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:19.900 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:19.899550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:30.010 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:30.008893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:30.020 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:30.019023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:40.123 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:40.122185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:40.134 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:40.133857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:50.248 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:50.247732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:05:50.262 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:05:50.261356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:00.365 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:00.364190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:00.378 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:00.377365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:10.480 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:10.479600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:10.493 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:10.491391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:20.622 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:20.621571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:20.633 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:20.632666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:25.568 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:25.567061 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T22:06:26.307 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:26.306306 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T22:06:26.295+0000 7fd1060fd380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T22:06:26.307 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T22:06:26.295+0000 7fd1060fd380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T22:06:26.308 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T22:06:26.295+0000 7fd1060fd380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T22:06:26.308 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T22:06:30.757 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:30.756798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:30.769 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:30.768137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:40.835 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:40.834465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:40.849 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:40.847939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:50.948 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:50.947810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:06:50.960 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:06:50.959432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:01.067 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:01.066372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:01.079 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:01.078317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:11.155 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:11.154914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:11.227 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:11.226967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:21.321 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:21.320414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:21.893 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:21.332664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:31.445 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:31.444374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:31.457 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:31.457056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:41.556 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:41.555183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:41.569 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:41.568539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:51.655 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:51.653792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:07:51.667 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:07:51.667038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:01.740 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:01.739682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:01.751 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:01.750982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:11.838 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:11.837322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:11.897 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:11.896436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:21.989 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:21.988080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:22.039 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:22.036540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:32.163 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:32.162306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:32.174 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:32.173648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:42.296 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:42.294943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:42.306 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:42.305341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:52.417 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:52.416212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:08:52.428 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:08:52.428079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:02.493 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:02.492329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:02.504 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:02.503328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:12.581 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:12.580360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:12.597 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:12.596937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:22.732 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:22.731382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:22.746 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:22.745218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:32.802 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:32.801207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:32.814 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:32.812946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:42.883 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:42.882055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:42.895 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:42.894629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:53.010 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:53.009706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:09:53.022 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:09:53.021470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:03.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:03.091524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:03.105 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:03.104191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:13.201 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:13.200236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:13.213 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:13.212574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:23.280 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:23.279808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:23.291 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:23.290680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:33.361 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:33.360022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:33.372 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:33.372014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:43.443 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:43.442426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:43.456 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:43.455252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:53.560 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:53.559215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:10:53.573 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:10:53.572725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:03.679 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:03.678576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:03.693 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:03.692203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:13.785 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:13.784175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:13.799 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:13.796875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:23.906 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:23.905605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:23.919 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:23.918007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:33.977 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:33.976677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:34.033 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:34.027883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:44.139 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:44.138081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:44.150 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:44.149288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:54.265 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:54.263372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:11:54.278 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:11:54.277901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:04.385 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:04.384649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:04.396 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:04.395428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:14.472 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:14.471727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:14.526 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:14.526214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:24.604 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:24.602890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:24.614 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:24.613722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:34.687 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:34.686290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:34.729 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:34.728027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:44.844 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:44.843740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:44.856 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:44.855822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:54.932 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:54.931101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:12:54.944 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:12:54.943778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:04.992 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:04.991332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:05.005 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:05.004028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:15.073 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:15.072635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:15.086 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:15.086056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:25.149 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:25.147849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:25.161 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:25.160917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:35.252 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:35.251349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:35.265 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:35.263132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:45.376 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:45.374885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:45.390 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:45.389278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:55.503 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:55.502361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:13:55.516 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:13:55.516060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:05.629 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:05.628244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:05.643 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:05.641790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:15.703 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:15.702293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:15.715 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:15.714679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:25.789 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:25.788336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:25.842 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:25.840909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:35.899 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:35.898954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:35.910 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:35.909841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:46.024 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:46.022704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:46.035 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:46.034593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:56.124 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:56.122862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:14:56.137 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:14:56.136139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:06.234 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:06.233352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:06.248 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:06.247474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:16.355 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:16.353944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:16.367 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:16.366857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:26.463 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:26.462542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:26.476 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:26.475704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:36.554 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:36.553402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:36.618 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:36.617267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:46.692 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:46.690220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:46.703 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:46.702355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:56.771 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:56.770249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:15:56.782 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:15:56.781021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:06.862 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:06.861731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:06.874 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:06.873358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:16.982 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:16.981455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:16.999 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:16.998516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:27.068 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:27.067691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:27.080 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:27.079633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:37.181 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:37.180356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:37.192 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:37.192147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:47.304 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:47.303379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:47.318 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:47.317346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:57.383 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:57.382213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:16:57.398 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:16:57.394326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:07.490 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:07.488919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:07.502 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:07.501315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:17.570 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:17.569739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:17.583 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:17.582228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:27.646 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:27.645914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:27.706 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:27.705967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:37.821 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:37.820934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:37.831 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:37.831006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:47.921 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:47.920155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:47.933 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:47.932932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:58.015 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:58.014023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:17:58.025 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:17:58.025118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:08.120 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:08.119891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:08.131 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:08.130507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:18.233 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:18.232407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:18.264 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:18.263277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:28.371 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:28.370171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:28.477 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:28.382422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:38.446 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:38.444963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:38.458 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:38.457316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:48.561 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:48.560134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:48.573 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:48.572169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:58.765 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:58.763908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:18:58.777 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:18:58.776994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:08.974 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:08.972711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:08.985 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:08.984976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:19.100 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:19.099647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:19.131 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:19.130309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:29.237 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:29.235961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:29.270 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:29.269091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:39.371 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:39.370462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:39.383 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:39.382600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:49.503 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:49.502505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:49.517 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:49.516693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:59.585 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:59.584485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:19:59.598 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:19:59.597304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:09.666 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:09.665201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:09.677 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:09.676995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:19.749 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:19.748751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:19.781 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:19.780470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:29.858 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:29.857046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:29.873 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:29.871685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:39.947 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:39.945740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:39.960 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:39.959654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:50.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:50.063051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:20:50.076 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:20:50.075630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:00.241 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:00.240069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:00.254 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:00.253961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:10.314 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:10.313403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:10.326 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:10.325326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:20.442 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:20.441156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:20.454 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:20.453885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:30.516 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:30.512361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:30.527 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:30.526298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:40.635 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:40.633647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:40.645 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:40.644758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:50.800 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:50.799006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:21:50.814 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:21:50.813632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:00.921 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:00.920818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:00.933 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:00.932706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:11.046 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:11.045406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:11.057 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:11.056691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:21.135 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:21.133806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:21.172 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:21.171153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:31.285 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:31.284287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:31.299 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:31.298597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:41.355 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:41.354175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:41.366 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:41.365531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:51.473 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:51.472029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:22:51.483 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:22:51.482930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:01.551 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:01.549769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:01.563 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:01.562526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:06.316 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:06.315476 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T22:23:06.988 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:06.987514 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T22:23:06.974+0000 7f98d37f9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T22:23:06.989 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T22:23:06.974+0000 7f98d37f9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T22:23:06.989 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T22:23:06.974+0000 7f98d37f9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T22:23:06.989 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T22:23:11.675 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:11.674003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:11.689 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:11.687907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:21.748 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:21.747580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:21.762 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:21.761456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:31.869 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:31.868363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:31.881 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:31.880771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:41.943 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:41.942662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:42.021 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:42.020468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:52.136 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:52.135339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:23:52.148 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:23:52.147577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:02.211 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:02.210564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:02.223 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:02.222316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:12.313 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:12.312310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:12.324 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:12.323383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:22.388 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:22.387739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:22.401 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:22.400538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:32.499 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:32.498862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:32.512 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:32.511690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:42.609 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:42.608809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:42.625 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:42.624323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:52.724 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:52.723520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:24:52.737 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:24:52.736864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:02.837 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:02.836570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:02.851 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:02.848252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:12.921 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:12.919889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:12.931 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:12.930365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:23.049 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:23.048487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:23.059 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:23.058994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:33.198 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:33.197418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:33.210 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:33.209774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:43.309 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:43.308333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:43.326 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:43.325365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:53.384 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:53.383330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:25:53.397 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:25:53.395372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:03.459 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:03.458340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:03.472 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:03.471573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:13.536 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:13.535514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:13.549 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:13.548345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:23.614 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:23.612373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:23.624 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:23.623439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:33.750 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:33.749040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:33.761 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:33.760509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:43.842 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:43.841296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:43.901 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:43.900118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:53.959 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:53.958120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:26:53.970 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:26:53.969864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:04.078 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:04.077079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:04.091 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:04.090167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:14.189 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:14.188054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:14.201 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:14.200461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:24.280 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:24.278787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:24.290 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:24.289475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:34.348 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:34.345833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:34.417 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:34.416801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:44.479 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:44.476802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:44.487 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:44.487070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:54.617 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:54.615945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:27:54.629 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:27:54.628522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:04.741 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:04.740682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:04.754 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:04.753587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:14.849 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:14.848291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:14.862 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:14.861306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:24.931 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:24.929620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:24.942 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:24.941529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:35.040 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:35.038992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:35.051 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:35.050711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:45.139 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:45.138211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:45.151 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:45.149931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:55.269 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:55.268277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:28:55.279 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:28:55.278669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:05.372 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:05.370824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:05.383 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:05.382008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:15.501 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:15.499792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:15.514 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:15.513365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:25.608 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:25.607395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:25.619 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:25.618896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:35.693 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:35.692764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:35.706 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:35.705454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:45.798 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:45.797075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:45.859 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:45.858978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:55.989 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:55.988161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:29:56.001 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:29:56.000776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:06.050 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:06.049440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:06.062 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:06.061124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:16.132 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:16.131474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:16.141 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:16.140694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:26.278 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:26.276700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:26.291 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:26.289950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:36.378 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:36.377607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:36.390 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:36.389100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:46.461 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:46.459904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:46.477 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:46.475830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:56.584 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:56.582817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:30:56.595 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:30:56.594501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:06.716 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:06.715323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:06.735 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:06.734781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:16.833 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:16.832623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:16.845 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:16.844014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:26.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:26.960301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:26.969 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:26.968897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:37.083 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:37.082430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:37.096 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:37.095104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:47.215 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:47.214675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:47.228 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:47.227007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:57.295 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:57.291687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:31:57.354 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:31:57.353286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:07.420 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:07.418843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:07.471 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:07.470273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:17.536 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:17.535805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:17.548 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:17.547252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:27.622 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:27.621106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:27.676 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:27.675333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:37.795 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:37.793920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:37.804 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:37.804000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:47.943 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:47.942070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:47.955 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:47.954512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:58.025 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:58.023459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:32:58.037 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:32:58.036416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:08.134 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:08.132858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:08.143 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:08.142166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:18.251 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:18.249362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:18.261 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:18.260815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:28.328 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:28.327331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:28.340 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:28.338738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:38.410 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:38.408928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:38.486 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:38.485074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:48.553 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:48.551953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:48.565 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:48.564691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:58.673 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:58.671628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:33:58.685 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:33:58.684354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:08.808 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:08.807286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:08.820 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:08.819987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:18.914 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:18.912915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:18.928 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:18.926742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:28.987 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:28.986523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:28.999 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:28.998722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:39.102 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:39.100324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:39.115 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:39.114474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:49.221 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:49.220490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:49.236 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:49.234747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:59.293 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:59.292075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:34:59.305 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:34:59.304695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:09.401 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:09.400253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:09.414 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:09.413927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:19.484 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:19.483367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:19.494 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:19.493233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:29.564 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:29.562948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:29.575 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:29.574684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:39.681 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:39.680321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:39.694 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:39.692909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:49.756 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:49.755024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:49.831 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:49.830672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:59.935 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:59.933840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:35:59.946 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:35:59.945906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:10.041 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:10.039681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:10.052 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:10.051627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:20.150 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:20.148066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:20.163 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:20.162331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:30.238 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:30.237459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:30.253 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:30.252977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:40.345 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:40.343682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:40.357 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:40.356063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:50.465 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:50.464043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:36:50.476 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:36:50.475881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:00.615 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:00.613743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:00.628 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:00.627500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:10.688 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:10.686975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:10.697 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:10.696964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:20.889 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:20.888589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:20.899 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:20.897972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:30.996 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:30.994936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:31.009 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:31.008259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:41.074 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:41.072919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:41.085 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:41.084641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:51.181 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:51.180356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:37:51.193 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:37:51.192753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:01.291 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:01.289583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:01.304 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:01.303383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:11.440 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:11.438923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:11.452 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:11.451841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:21.532 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:21.531417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:21.544 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:21.543320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:31.623 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:31.622610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:31.637 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:31.636032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:41.738 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:41.737425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:41.749 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:41.748594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:51.813 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:51.803323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:38:51.817 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:38:51.816246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:01.914 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:01.912969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:01.927 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:01.925982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:11.991 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:11.989872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:12.003 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:12.002353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:22.107 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:22.105862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:22.120 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:22.118240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:32.176 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:32.175267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:32.188 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:32.187742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:42.285 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:42.284441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:42.308 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:42.307090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:46.996 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:46.995372 I | op-mon: parsing mon endpoints: b=172.21.15.93:6789,c=172.21.15.111:6789,a=172.21.15.161:6789 2022-03-10T22:39:47.821 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:47.819503 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T22:39:47.808+0000 7fa036bf8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T22:39:47.821 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T22:39:47.808+0000 7fa036bf8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T22:39:47.821 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10T22:39:47.808+0000 7fa036bf8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T22:39:47.821 INFO:tasks.rook.operator.smithi093.stdout:couldn't init storage provider. : exit status 5 2022-03-10T22:39:52.542 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:52.541493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:39:52.555 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:39:52.554036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:02.617 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:02.616314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:02.632 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:02.631290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:12.751 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:12.749893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:12.763 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:12.762663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:22.828 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:22.827036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:22.841 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:22.840598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:32.950 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:32.948786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:32.961 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:32.960714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:43.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:43.063569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:43.078 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:43.077448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:53.153 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:53.152648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:40:53.165 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:40:53.164102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:03.281 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:03.280464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:03.299 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:03.299028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:13.408 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:13.407007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:13.423 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:13.422446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:23.547 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:23.545847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:23.557 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:23.556833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:33.634 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:33.633272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:33.646 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:33.645836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:43.743 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:43.742444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:43.757 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:43.756242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:53.820 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:53.819286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:41:53.833 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:41:53.832776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:03.897 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:03.896232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:03.909 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:03.908809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:13.974 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:13.973496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:13.988 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:13.986972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:24.064 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:24.063445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:24.078 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:24.076495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:34.194 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:34.193348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:34.209 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:34.208162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:44.316 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:44.315337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:44.328 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:44.327554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:54.437 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:54.436048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:42:54.450 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:42:54.449469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:04.545 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:04.543870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:04.559 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:04.558385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:14.664 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:14.662931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:14.676 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:14.675733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:24.741 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:24.740254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:24.754 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:24.752769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:34.852 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:34.851487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:34.865 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:34.864299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:44.929 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:44.928164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:44.940 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:44.939617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:55.006 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:55.005456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:43:55.020 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:43:55.019081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:05.145 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:05.143731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:05.156 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:05.155801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:15.229 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:15.227879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:15.267 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:15.263208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:25.330 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:25.329551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:25.343 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:25.341940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:35.415 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:35.413755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:35.427 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:35.426523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:45.505 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:45.504178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:45.517 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:45.516361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:55.745 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:55.743977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:44:55.758 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:44:55.757121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:05.911 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:05.909820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:05.922 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:05.921778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:16.010 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:16.008742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:16.019 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:16.018772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:26.123 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:26.121826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:26.135 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:26.134097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:36.203 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:36.201817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:36.256 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:36.213038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:46.316 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:46.314428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:46.326 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:46.325763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:56.393 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:56.392167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:45:56.406 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:45:56.405611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:06.469 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:06.468263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:06.519 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:06.515335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:16.578 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:16.576896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:16.589 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:16.587261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:26.653 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:26.652563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:26.665 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:26.664222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:36.720 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:36.718170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:36.730 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:36.728534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:46.834 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:46.832625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:46.847 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:46.846106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:56.962 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:56.961277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:46:56.976 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:46:56.975339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:07.092 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:07.091282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:07.103 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:07.102629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:17.175 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:17.173556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:17.186 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:17.185219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:27.330 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:27.329312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:27.339 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:27.338370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:37.417 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:37.415624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:37.429 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:37.428354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:47.497 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:47.495804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:47.509 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:47.507968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:57.642 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:57.640750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:47:57.655 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:47:57.653838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:48:07.725 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:48:07.723671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:48:07.737 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:48:07.736207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:48:17.813 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:48:17.811737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:48:17.824 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:48:17.822978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:48:27.999 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:48:27.991345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T22:48:28.011 INFO:tasks.rook.operator.smithi093.stdout:2022-03-10 22:48:28.010847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-11T01:08:53.002 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-11T01:08:53.041 DEBUG:teuthology.task.console_log:Killing console logger for smithi093 2022-03-11T01:08:53.042 DEBUG:teuthology.task.console_log:Killing console logger for smithi111 2022-03-11T01:08:53.044 DEBUG:teuthology.task.console_log:Killing console logger for smithi161 2022-03-11T01:08:53.044 DEBUG:teuthology.exit:Finished running handlers