2022-02-21T16:58:05.228 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-21T16:58:05.230 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698384 --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698384/orig.config.yaml 2022-02-21T16:58:05.276 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-02-21T16:58:05.361 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698384 branch: wip-yuri4-testing-2022-02-18-0800 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} email: ceph-qa@ceph.io first_in_suite: false job_id: '6698384' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri4-testing-2022-02-18-0800 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri4-testing-2022-02-18-0800 sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e owner: scheduled_yuriw@teuthology priority: 79 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri4-testing-2022-02-18-0800 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e targets: smithi005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDow04xMTKgeNXkQceZJIvn/5U275Hr/uUB7fGj6P3K4Qi0NmUvQ/ltBUiVMaX5uEeTBSgXcE79D6Xj4wgQHPLAMhN4cUhvr0O7jP313mWn28YLOEckdRgAK68hWpYkQkdHT02gBk+indxaTcdn+b6ojb0ZbrmQoII6Lj+MpfWbPE7shsiD/h5BVxxK7kTWgzEJuuYdICD7RdihdbDvfLWaNBEmzHJ5HVHFzit5xpYKf5iBlnjHd7ApMnjT3vYKcelkM4zne2GUJFc5aupM/YKhohv/gt4+wNhVAEMPwYsJfPsY+R2vhQH78RvXCBE9g+ZKOW/Kun3bFfziz2L516U82EogFTMJuGWZgc5AXDU8+yTo59DGA8C60duAYSl08iltYifHqrtOgluyD07prdl9GtF5UbDWPkIYKhyxApklrg5EuJxNjYZ2LFqbeDAsc9gpA/V208bB1PoKWDybC4TQEPosiO0mXDbaY/nqM6sGTqGcdD3KKt5CVUE0k9/9uIE= smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgIUF/s2swRNicgKdp1Fi/R94MO7VNkG8L6pVH0+L+XBdNfJmYrZOzacZzBny/zD5zZmiqv2XebDueGpxxaYrTT4Cb1DKx4EguMgqtcGl1EtLGyEt1lgZ0ICCy2DFtlfS4/d92DPjNsHOjrMfSM1wD6F/ubcaFLYVaOnSLd7hQ2bNRANCTr9D7lXpRhNYFOCm9dfTTLwEWqC4E7Lvo0LeoLEK10TO1WuwqI/4khxoNIsqUUUz7xR37f+uLp7evbqH4cbitM2hXa8nF5MaIBop+mx01u/+3NYKiJpjyNvPJ7OE0cTJ5ZB5lhxH9kSUrl4J9EygwyFaoNBi1y/qwTunmUXG2D8C1QetTcTc3/4WjxjpQvS2OxmrmtHkHdP3toKfVlZJCgB4N4DKYRiO6d2eVardUvRijsRpVJWZm7KTKZdl2GwGEz8G5kUhY1IKdhU5EbMp1QO0kK+kmyNXvl1G4vFJ4DvBJDigyQ7xD8OqOvcDj+LYLIj8EP9plYiqDT1M= smithi067.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZ8/QJjcfFvp1h9MCkD/yrpIJbEThGmc46+xzudOdEX28RBzxey6MElgUBLLfKHVW17KNfQSmdmMkC1QnU3anPE0n66z9hxmEpcOE/vAmRWi5+yJTXa2fbEIBrWmlhJhtGjPs9nvtaqDQTL3hPIllNPJ/XZN39UDFC0YnWx5wbhl92xTkdlfK37jC2Je8DVbI8qm8R8QsYA571vL77/IgoZJNyrXHpnbtucIwWjvqtcSoio6NBe6Vr+8WVeVkj9SKODvFdXIXm95tWclOC4er3N9Ep8PN4r0rbPqPXae0R3wWHczjaxq+S350Od7X5LAM/+JbOoF62ca4Mb1QJPor24BnfEIsSiP+CbJSBkOaJ8ZvkhmRzGn7HvTbnabuGFpU22FlisiLEc1V5gBU+PlwjVRzaVXLkDZ9J+mmNcLFdlYuS3zjRn+6CZNMLcOJlsBwi8UrFfSlZHegcNrCt9ulPuc6JA6nul0wuLoAPEX7PeGkfRUNTnclhMFtt8X+EOWM= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eea2521245e542c7c1a063d296779f572aa0255a timestamp: 2022-02-21_15:40:41 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 2022-02-21T16:58:05.362 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa; will attempt to use it 2022-02-21T16:58:05.363 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks 2022-02-21T16:58:05.363 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-02-21T16:58:05.367 INFO:teuthology.task.internal:Checking packages... 2022-02-21T16:58:05.390 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b05010b68efc838fdc35b41106e2ebdabe395a3e' 2022-02-21T16:58:05.391 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-02-21T16:58:05.392 INFO:teuthology.packaging:ref: None 2022-02-21T16:58:05.392 INFO:teuthology.packaging:tag: None 2022-02-21T16:58:05.393 INFO:teuthology.packaging:branch: wip-yuri4-testing-2022-02-18-0800 2022-02-21T16:58:05.393 INFO:teuthology.packaging:sha1: b05010b68efc838fdc35b41106e2ebdabe395a3e 2022-02-21T16:58:05.394 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri4-testing-2022-02-18-0800 2022-02-21T16:58:05.723 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-10781-gb05010b6-1focal 2022-02-21T16:58:05.724 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-02-21T16:58:05.732 INFO:teuthology.task.internal:no buildpackages task found 2022-02-21T16:58:05.732 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-02-21T16:58:05.777 INFO:teuthology.task.internal:Saving configuration 2022-02-21T16:58:05.790 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-02-21T16:58:05.829 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-21T16:58:05.853 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698384', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-02-21 16:52:26.830294', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG2DwT3K1GOcpAoN5SY0jt58cG2sDi4gaz5GqoNz5K8j'} 2022-02-21T16:58:05.876 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698384', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-02-21 16:52:26.828969', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAOxgjMXlRa8JKHPSh5KVMN6f/iQJxunInspF8mfiiLK'} 2022-02-21T16:58:05.899 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698384', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-02-21 16:52:26.831300', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHhCIBY4qt0/VGl9qQo9/oIcUyXyOnc3fIzmfWDR8y4B'} 2022-02-21T16:58:05.899 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-02-21T16:58:05.908 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-02-21T16:58:05.908 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-02-21T16:58:05.908 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-02-21T16:58:05.909 INFO:teuthology.run_tasks:Running task console_log... 2022-02-21T16:58:06.030 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe219b9dea0>, signals=[15]) 2022-02-21T16:58:06.032 INFO:teuthology.run_tasks:Running task internal.connect... 2022-02-21T16:58:06.040 INFO:teuthology.task.internal:Opening connections... 2022-02-21T16:58:06.041 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2022-02-21T16:58:06.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T16:58:06.108 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2022-02-21T16:58:06.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T16:58:06.163 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2022-02-21T16:58:06.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T16:58:06.219 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-02-21T16:58:06.234 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2022-02-21T16:58:06.242 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2022-02-21T16:58:06.243 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2022-02-21T16:58:06.292 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="Ubuntu" 2022-02-21T16:58:06.292 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-21T16:58:06.293 INFO:teuthology.orchestra.run.smithi005.stdout:ID=ubuntu 2022-02-21T16:58:06.293 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE=debian 2022-02-21T16:58:06.293 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-21T16:58:06.294 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="20.04" 2022-02-21T16:58:06.294 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-21T16:58:06.294 INFO:teuthology.orchestra.run.smithi005.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-21T16:58:06.294 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-21T16:58:06.295 INFO:teuthology.orchestra.run.smithi005.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-21T16:58:06.295 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_CODENAME=focal 2022-02-21T16:58:06.295 INFO:teuthology.orchestra.run.smithi005.stdout:UBUNTU_CODENAME=focal 2022-02-21T16:58:06.296 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2022-02-21T16:58:06.319 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2022-02-21T16:58:06.326 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2022-02-21T16:58:06.327 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2022-02-21T16:58:06.376 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="Ubuntu" 2022-02-21T16:58:06.376 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-21T16:58:06.376 INFO:teuthology.orchestra.run.smithi033.stdout:ID=ubuntu 2022-02-21T16:58:06.377 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE=debian 2022-02-21T16:58:06.377 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-21T16:58:06.377 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="20.04" 2022-02-21T16:58:06.377 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-21T16:58:06.378 INFO:teuthology.orchestra.run.smithi033.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-21T16:58:06.378 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-21T16:58:06.378 INFO:teuthology.orchestra.run.smithi033.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-21T16:58:06.378 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_CODENAME=focal 2022-02-21T16:58:06.379 INFO:teuthology.orchestra.run.smithi033.stdout:UBUNTU_CODENAME=focal 2022-02-21T16:58:06.379 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2022-02-21T16:58:06.398 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2022-02-21T16:58:06.405 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2022-02-21T16:58:06.405 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2022-02-21T16:58:06.454 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Ubuntu" 2022-02-21T16:58:06.454 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-21T16:58:06.455 INFO:teuthology.orchestra.run.smithi067.stdout:ID=ubuntu 2022-02-21T16:58:06.455 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE=debian 2022-02-21T16:58:06.455 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-21T16:58:06.455 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="20.04" 2022-02-21T16:58:06.456 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-21T16:58:06.456 INFO:teuthology.orchestra.run.smithi067.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-21T16:58:06.458 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-21T16:58:06.458 INFO:teuthology.orchestra.run.smithi067.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-21T16:58:06.459 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_CODENAME=focal 2022-02-21T16:58:06.459 INFO:teuthology.orchestra.run.smithi067.stdout:UBUNTU_CODENAME=focal 2022-02-21T16:58:06.459 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2022-02-21T16:58:06.477 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-02-21T16:58:06.488 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-02-21T16:58:06.510 INFO:teuthology.task.internal:Checking for old test directory... 2022-02-21T16:58:06.511 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2022-02-21T16:58:06.514 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2022-02-21T16:58:06.520 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2022-02-21T16:58:06.528 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-02-21T16:58:06.536 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-02-21T16:58:06.536 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2022-02-21T16:58:06.564 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2022-02-21T16:58:06.566 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2022-02-21T16:58:06.600 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-02-21T16:58:06.684 INFO:teuthology.run_tasks:Running task kernel... 2022-02-21T16:58:06.700 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-02-21T16:58:06.701 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-02-21T16:58:06.701 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-02-21T16:58:06.704 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-02-21T16:58:06.705 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2022-02-21T16:58:06.706 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2022-02-21T16:58:06.707 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2022-02-21T16:58:06.715 INFO:teuthology.orchestra.run.smithi005.stdout:5.4.0-97-generic 2022-02-21T16:58:06.715 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.4.0-97-generic 2022-02-21T16:58:06.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2022-02-21T16:58:06.717 INFO:teuthology.orchestra.run.smithi033.stdout:5.4.0-97-generic 2022-02-21T16:58:06.718 INFO:teuthology.orchestra.run.smithi067.stdout:5.4.0-97-generic 2022-02-21T16:58:06.718 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.4.0-97-generic 2022-02-21T16:58:06.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2022-02-21T16:58:06.720 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.4.0-97-generic 2022-02-21T16:58:06.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2022-02-21T16:58:06.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2022-02-21T16:58:06.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2022-02-21T16:58:06.845 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2022-02-21T16:58:06.978 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-02-21T16:58:07.005 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-02-21T16:58:07.089 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T16:58:07.113 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T16:58:07.135 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T16:58:07.136 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-02-21T16:58:07.172 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-02-21T16:58:07.196 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-02-21T16:58:07.201 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-02-21T16:58:07.281 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,247 kB] 2022-02-21T16:58:07.295 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,247 kB] 2022-02-21T16:58:07.410 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [379 kB] 2022-02-21T16:58:07.414 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [219 kB] 2022-02-21T16:58:07.417 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,556 B] 2022-02-21T16:58:07.418 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-02-21T16:58:07.419 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [757 kB] 2022-02-21T16:58:07.429 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [379 kB] 2022-02-21T16:58:07.433 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [108 kB] 2022-02-21T16:58:07.433 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [219 kB] 2022-02-21T16:58:07.434 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-02-21T16:58:07.435 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [678 kB] 2022-02-21T16:58:07.440 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,556 B] 2022-02-21T16:58:07.441 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-02-21T16:58:07.442 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [757 kB] 2022-02-21T16:58:07.450 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [534 kB] 2022-02-21T16:58:07.460 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [116 kB] 2022-02-21T16:58:07.460 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.1 kB] 2022-02-21T16:58:07.463 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [108 kB] 2022-02-21T16:58:07.465 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-02-21T16:58:07.466 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [678 kB] 2022-02-21T16:58:07.474 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [534 kB] 2022-02-21T16:58:07.483 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [116 kB] 2022-02-21T16:58:07.484 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.1 kB] 2022-02-21T16:58:07.487 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-02-21T16:58:07.488 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-02-21T16:58:07.488 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-02-21T16:58:07.488 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-02-21T16:58:07.497 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-02-21T16:58:07.498 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-02-21T16:58:07.498 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-02-21T16:58:07.498 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-02-21T16:58:07.515 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-02-21T16:58:07.549 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-02-21T16:58:07.612 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-02-21T16:58:07.618 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,247 kB] 2022-02-21T16:58:07.641 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [607 kB] 2022-02-21T16:58:07.681 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [607 kB] 2022-02-21T16:58:07.779 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [607 kB] 2022-02-21T16:58:07.882 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,598 kB] 2022-02-21T16:58:07.937 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,598 kB] 2022-02-21T16:58:07.981 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,598 kB] 2022-02-21T16:58:08.032 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [379 kB] 2022-02-21T16:58:08.044 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [306 kB] 2022-02-21T16:58:08.062 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.7 kB] 2022-02-21T16:58:08.063 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [818 kB] 2022-02-21T16:58:08.096 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [306 kB] 2022-02-21T16:58:08.116 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-02-21T16:58:08.117 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [116 kB] 2022-02-21T16:58:08.118 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.7 kB] 2022-02-21T16:58:08.118 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-02-21T16:58:08.119 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [818 kB] 2022-02-21T16:58:08.122 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-02-21T16:58:08.122 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [668 kB] 2022-02-21T16:58:08.147 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [904 kB] 2022-02-21T16:58:08.149 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [219 kB] 2022-02-21T16:58:08.155 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [116 kB] 2022-02-21T16:58:08.161 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-02-21T16:58:08.161 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [668 kB] 2022-02-21T16:58:08.171 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,556 B] 2022-02-21T16:58:08.172 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-02-21T16:58:08.173 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [757 kB] 2022-02-21T16:58:08.176 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-02-21T16:58:08.183 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.1 kB] 2022-02-21T16:58:08.194 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [904 kB] 2022-02-21T16:58:08.219 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [306 kB] 2022-02-21T16:58:08.222 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-02-21T16:58:08.222 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-02-21T16:58:08.223 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-02-21T16:58:08.223 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-02-21T16:58:08.224 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-02-21T16:58:08.224 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [11.7 kB] 2022-02-21T16:58:08.228 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.1 kB] 2022-02-21T16:58:08.245 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [21.6 kB] 2022-02-21T16:58:08.246 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.0 kB] 2022-02-21T16:58:08.247 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.7 kB] 2022-02-21T16:58:08.247 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [108 kB] 2022-02-21T16:58:08.248 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-02-21T16:58:08.248 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-02-21T16:58:08.248 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [534 kB] 2022-02-21T16:58:08.249 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [818 kB] 2022-02-21T16:58:08.253 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [678 kB] 2022-02-21T16:58:08.255 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-02-21T16:58:08.255 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-02-21T16:58:08.256 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-02-21T16:58:08.257 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-02-21T16:58:08.257 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [11.7 kB] 2022-02-21T16:58:08.282 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [116 kB] 2022-02-21T16:58:08.290 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.1 kB] 2022-02-21T16:58:08.303 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [21.6 kB] 2022-02-21T16:58:08.303 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.0 kB] 2022-02-21T16:58:08.313 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [116 kB] 2022-02-21T16:58:08.344 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [716 B] 2022-02-21T16:58:08.349 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-02-21T16:58:08.350 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-02-21T16:58:08.351 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-02-21T16:58:08.351 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-02-21T16:58:08.397 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [716 B] 2022-02-21T16:58:08.411 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-02-21T16:58:08.411 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [668 kB] 2022-02-21T16:58:08.426 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [904 kB] 2022-02-21T16:58:08.464 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-02-21T16:58:08.471 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.1 kB] 2022-02-21T16:58:08.471 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-02-21T16:58:08.472 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-02-21T16:58:08.472 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-02-21T16:58:08.472 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-02-21T16:58:08.494 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [11.7 kB] 2022-02-21T16:58:08.495 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [21.6 kB] 2022-02-21T16:58:08.577 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.0 kB] 2022-02-21T16:58:08.577 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [716 B] 2022-02-21T16:58:10.744 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 9,819 kB in 2s (5,653 kB/s) 2022-02-21T16:58:10.915 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 9,819 kB in 2s (5,796 kB/s) 2022-02-21T16:58:11.316 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 9,819 kB in 2s (4,598 kB/s) 2022-02-21T16:58:11.969 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-02-21T16:58:11.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T16:58:12.065 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-02-21T16:58:12.071 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2022-02-21T16:58:12.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T16:58:12.160 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2022-02-21T16:58:12.244 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2022-02-21T16:58:12.245 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2022-02-21T16:58:12.336 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2022-02-21T16:58:12.337 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2022-02-21T16:58:12.397 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2022-02-21T16:58:12.397 INFO:teuthology.orchestra.run.smithi033.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T16:58:12.397 INFO:teuthology.orchestra.run.smithi033.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-02-21T16:58:12.398 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-02-21T16:58:12.398 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2022-02-21T16:58:12.399 INFO:teuthology.orchestra.run.smithi033.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-02-21T16:58:12.399 INFO:teuthology.orchestra.run.smithi033.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-02-21T16:58:12.441 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2022-02-21T16:58:12.442 INFO:teuthology.orchestra.run.smithi033.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T16:58:12.442 INFO:teuthology.orchestra.run.smithi033.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-02-21T16:58:12.443 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-02-21T16:58:12.443 INFO:teuthology.orchestra.run.smithi033.stdout: thermald upower usbmuxd 2022-02-21T16:58:12.484 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2022-02-21T16:58:12.484 INFO:teuthology.orchestra.run.smithi067.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T16:58:12.485 INFO:teuthology.orchestra.run.smithi067.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-02-21T16:58:12.485 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-02-21T16:58:12.485 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2022-02-21T16:58:12.486 INFO:teuthology.orchestra.run.smithi067.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-02-21T16:58:12.486 INFO:teuthology.orchestra.run.smithi067.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-02-21T16:58:12.538 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2022-02-21T16:58:12.538 INFO:teuthology.orchestra.run.smithi067.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T16:58:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-02-21T16:58:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-02-21T16:58:12.539 INFO:teuthology.orchestra.run.smithi067.stdout: thermald upower usbmuxd 2022-02-21T16:58:12.589 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-02-21T16:58:12.615 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T16:58:12.685 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-02-21T16:58:12.718 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 13 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T16:58:12.719 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 80.1 MB of archives. 2022-02-21T16:58:12.719 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 409 MB of additional disk space will be used. 2022-02-21T16:58:12.719 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-02-21T16:58:12.778 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 13 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T16:58:12.779 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 80.1 MB of archives. 2022-02-21T16:58:12.779 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 409 MB of additional disk space will be used. 2022-02-21T16:58:12.779 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-02-21T16:58:12.864 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2022-02-21T16:58:12.864 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2022-02-21T16:58:13.017 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2022-02-21T16:58:13.017 INFO:teuthology.orchestra.run.smithi005.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T16:58:13.018 INFO:teuthology.orchestra.run.smithi005.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-02-21T16:58:13.018 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-02-21T16:58:13.019 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2022-02-21T16:58:13.019 INFO:teuthology.orchestra.run.smithi005.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-02-21T16:58:13.019 INFO:teuthology.orchestra.run.smithi005.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-02-21T16:58:13.051 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-02-21T16:58:13.063 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2022-02-21T16:58:13.064 INFO:teuthology.orchestra.run.smithi005.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-02-21T16:58:13.064 INFO:teuthology.orchestra.run.smithi005.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-02-21T16:58:13.064 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-02-21T16:58:13.064 INFO:teuthology.orchestra.run.smithi005.stdout: thermald upower usbmuxd 2022-02-21T16:58:13.079 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-02-21T16:58:13.104 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-02-21T16:58:13.158 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-02-21T16:58:13.180 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-02-21T16:58:13.190 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-02-21T16:58:13.204 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-02-21T16:58:13.242 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-02-21T16:58:13.252 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 13 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T16:58:13.252 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 80.1 MB of archives. 2022-02-21T16:58:13.252 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 409 MB of additional disk space will be used. 2022-02-21T16:58:13.253 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-02-21T16:58:13.289 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-02-21T16:58:13.345 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-02-21T16:58:13.545 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-02-21T16:58:13.577 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-02-21T16:58:13.578 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-02-21T16:58:13.680 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-02-21T16:58:13.689 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-02-21T16:58:14.584 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-02-21T16:58:14.906 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-02-21T16:58:15.159 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-02-21T16:58:15.236 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-02-21T16:58:15.363 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-02-21T16:58:15.788 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-02-21T16:58:18.148 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-02-21T16:58:18.148 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-02-21T16:58:18.149 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-02-21T16:58:18.154 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-02-21T16:58:18.155 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-02-21T16:58:18.613 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 80.1 MB in 5s (15.6 MB/s) 2022-02-21T16:58:18.778 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-02-21T16:58:18.851 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-02-21T16:58:18.851 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-02-21T16:58:18.852 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-02-21T16:58:18.858 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-02-21T16:58:18.860 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-02-21T16:58:19.088 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-02-21T16:58:19.088 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-02-21T16:58:19.089 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-02-21T16:58:19.095 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-02-21T16:58:19.097 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-02-21T16:58:19.287 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 80.1 MB in 6s (12.5 MB/s) 2022-02-21T16:58:19.491 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-02-21T16:58:19.521 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 80.1 MB in 7s (12.2 MB/s) 2022-02-21T16:58:19.712 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-02-21T16:58:20.276 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-02-21T16:58:20.281 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-02-21T16:58:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T16:58:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libplist3:amd64. 2022-02-21T16:58:20.827 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-02-21T16:58:20.870 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T16:58:20.966 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-02-21T16:58:20.974 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-02-21T16:58:21.114 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T16:58:21.159 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-02-21T16:58:21.165 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-02-21T16:58:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-02-21T16:58:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-02-21T16:58:21.264 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T16:58:21.292 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T16:58:21.500 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libplist3:amd64. 2022-02-21T16:58:21.514 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-02-21T16:58:21.550 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T16:58:21.609 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-02-21T16:58:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-02-21T16:58:21.659 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T16:58:21.678 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libplist3:amd64. 2022-02-21T16:58:21.692 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-02-21T16:58:21.737 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T16:58:21.853 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-02-21T16:58:21.866 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-02-21T16:58:21.903 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T16:58:21.980 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-02-21T16:58:21.995 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-02-21T16:58:22.015 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-02-21T16:58:22.029 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-02-21T16:58:22.056 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T16:58:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T16:58:22.189 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-02-21T16:58:22.204 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-02-21T16:58:22.248 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T16:58:22.384 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-02-21T16:58:22.399 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-02-21T16:58:22.442 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T16:58:22.506 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-02-21T16:58:22.521 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T16:58:22.565 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:22.568 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-02-21T16:58:22.580 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-02-21T16:58:22.609 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T16:58:22.812 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-02-21T16:58:22.825 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-02-21T16:58:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T16:58:22.920 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-02-21T16:58:22.936 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T16:58:22.979 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:23.182 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-02-21T16:58:23.197 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T16:58:23.232 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:24.824 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-02-21T16:58:24.840 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T16:58:24.937 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:25.372 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-02-21T16:58:25.388 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T16:58:25.452 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-02-21T16:58:25.465 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-02-21T16:58:25.466 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T16:58:25.480 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T16:58:25.483 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:25.510 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:25.688 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:25.977 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-02-21T16:58:25.990 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T16:58:26.025 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:26.285 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-02-21T16:58:26.298 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-02-21T16:58:26.335 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:31.735 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-02-21T16:58:31.751 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-02-21T16:58:31.785 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T16:58:31.953 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-02-21T16:58:31.968 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-02-21T16:58:32.011 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T16:58:32.116 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libevdev2:amd64. 2022-02-21T16:58:32.129 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-02-21T16:58:32.172 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T16:58:32.268 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-02-21T16:58:32.286 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-02-21T16:58:32.318 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T16:58:32.339 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libevdev2:amd64. 2022-02-21T16:58:32.356 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-02-21T16:58:32.398 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T16:58:32.589 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package thermald. 2022-02-21T16:58:32.603 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-02-21T16:58:32.643 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libevdev2:amd64. 2022-02-21T16:58:32.648 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T16:58:32.657 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-02-21T16:58:32.700 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T16:58:32.713 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package thermald. 2022-02-21T16:58:32.729 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-02-21T16:58:32.771 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T16:58:33.077 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package thermald. 2022-02-21T16:58:33.094 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-02-21T16:58:33.136 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T16:58:33.236 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package upower. 2022-02-21T16:58:33.250 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-02-21T16:58:33.301 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package upower. 2022-02-21T16:58:33.304 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking upower (0.99.11-1build2) ... 2022-02-21T16:58:33.315 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-02-21T16:58:33.369 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking upower (0.99.11-1build2) ... 2022-02-21T16:58:33.685 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package upower. 2022-02-21T16:58:33.698 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-02-21T16:58:33.709 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package usbmuxd. 2022-02-21T16:58:33.725 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-02-21T16:58:33.732 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package usbmuxd. 2022-02-21T16:58:33.745 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking upower (0.99.11-1build2) ... 2022-02-21T16:58:33.747 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-02-21T16:58:33.767 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T16:58:33.808 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T16:58:34.079 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T16:58:34.111 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T16:58:34.168 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package usbmuxd. 2022-02-21T16:58:34.181 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-02-21T16:58:34.227 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T16:58:34.227 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T16:58:34.233 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T16:58:34.352 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T16:58:34.375 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T16:58:34.501 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T16:58:34.509 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-02-21T16:58:34.556 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T16:58:34.628 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T16:58:34.656 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-02-21T16:58:34.762 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T16:58:34.774 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-02-21T16:58:34.888 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T16:58:34.900 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-02-21T16:58:34.921 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up upower (0.99.11-1build2) ... 2022-02-21T16:58:35.014 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T16:58:35.026 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-02-21T16:58:35.135 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-02-21T16:58:35.156 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up upower (0.99.11-1build2) ... 2022-02-21T16:58:35.261 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up upower (0.99.11-1build2) ... 2022-02-21T16:58:35.390 INFO:teuthology.orchestra.run.smithi033.stdout:upower.service is a disabled or a static unit, not starting it. 2022-02-21T16:58:35.456 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T16:58:35.614 INFO:teuthology.orchestra.run.smithi005.stdout:upower.service is a disabled or a static unit, not starting it. 2022-02-21T16:58:35.617 INFO:teuthology.orchestra.run.smithi033.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-02-21T16:58:35.631 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user `usbmux' (UID 114) ... 2022-02-21T16:58:35.631 INFO:teuthology.orchestra.run.smithi033.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-02-21T16:58:35.658 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T16:58:35.691 INFO:teuthology.orchestra.run.smithi067.stdout:upower.service is a disabled or a static unit, not starting it. 2022-02-21T16:58:35.738 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-02-21T16:58:35.824 INFO:teuthology.orchestra.run.smithi005.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-02-21T16:58:35.837 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user `usbmux' (UID 114) ... 2022-02-21T16:58:35.837 INFO:teuthology.orchestra.run.smithi005.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-02-21T16:58:35.916 INFO:teuthology.orchestra.run.smithi067.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-02-21T16:58:35.926 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user `usbmux' (UID 114) ... 2022-02-21T16:58:35.926 INFO:teuthology.orchestra.run.smithi067.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-02-21T16:58:36.140 INFO:teuthology.orchestra.run.smithi033.stdout:Not creating home directory `/var/lib/usbmux'. 2022-02-21T16:58:36.341 INFO:teuthology.orchestra.run.smithi005.stdout:Not creating home directory `/var/lib/usbmux'. 2022-02-21T16:58:36.363 INFO:teuthology.orchestra.run.smithi033.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-02-21T16:58:36.411 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T16:58:36.571 INFO:teuthology.orchestra.run.smithi005.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-02-21T16:58:36.630 INFO:teuthology.orchestra.run.smithi067.stdout:Not creating home directory `/var/lib/usbmux'. 2022-02-21T16:58:36.638 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T16:58:36.739 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T16:58:36.739 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T16:58:36.837 INFO:teuthology.orchestra.run.smithi067.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-02-21T16:58:36.884 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-02-21T16:58:36.955 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T16:58:36.956 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T16:58:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T16:58:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-02-21T16:58:37.223 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:37.426 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:37.664 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:38.505 INFO:teuthology.orchestra.run.smithi033.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-02-21T16:58:38.506 INFO:teuthology.orchestra.run.smithi033.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-02-21T16:58:38.700 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:38.740 INFO:teuthology.orchestra.run.smithi005.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-02-21T16:58:38.741 INFO:teuthology.orchestra.run.smithi005.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-02-21T16:58:38.894 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:38.957 INFO:teuthology.orchestra.run.smithi067.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-02-21T16:58:38.957 INFO:teuthology.orchestra.run.smithi067.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-02-21T16:58:39.107 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:40.052 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:40.459 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:40.487 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:41.428 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T16:58:41.554 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-02-21T16:58:41.847 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T16:58:41.990 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-02-21T16:58:42.753 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-02-21T16:58:42.912 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-02-21T16:58:46.370 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T16:58:46.859 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T16:58:46.917 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T16:58:47.005 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-02-21T16:58:47.122 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:47.322 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-02-21T16:58:47.322 INFO:teuthology.orchestra.run.smithi005.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-02-21T16:58:47.528 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-02-21T16:58:47.628 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-02-21T16:58:47.628 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:47.728 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-02-21T16:58:47.824 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-02-21T16:58:47.824 INFO:teuthology.orchestra.run.smithi033.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-02-21T16:58:47.920 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-02-21T16:58:47.921 INFO:teuthology.orchestra.run.smithi067.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-02-21T16:59:11.376 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-02-21T16:59:11.376 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub' 2022-02-21T16:59:11.387 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T16:59:11.409 INFO:teuthology.orchestra.run.smithi005.stdout:Generating grub configuration file ... 2022-02-21T16:59:11.735 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-02-21T16:59:11.736 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub' 2022-02-21T16:59:11.745 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T16:59:11.768 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T16:59:11.772 INFO:teuthology.orchestra.run.smithi033.stdout:Generating grub configuration file ... 2022-02-21T16:59:11.775 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-02-21T16:59:11.775 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub' 2022-02-21T16:59:11.780 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T16:59:11.793 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T16:59:11.804 INFO:teuthology.orchestra.run.smithi067.stdout:Generating grub configuration file ... 2022-02-21T16:59:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T16:59:12.137 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T16:59:12.142 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T16:59:12.179 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T16:59:12.193 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T16:59:12.205 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T16:59:12.533 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T16:59:12.545 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T16:59:12.590 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T16:59:12.602 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T16:59:14.822 INFO:teuthology.orchestra.run.smithi005.stdout:done 2022-02-21T16:59:14.929 INFO:teuthology.orchestra.run.smithi067.stdout:done 2022-02-21T16:59:15.146 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T16:59:15.174 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T16:59:15.175 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2022-02-21T16:59:15.175 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2022-02-21T16:59:15.175 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2022-02-21T16:59:15.175 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 19 2022-02-21T16:59:15.176 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T16:59:15.176 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2022-02-21T16:59:15.176 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta-hwe-5.13 2022-02-21T16:59:15.177 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T16:59:15.177 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T16:59:15.177 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T16:59:15.177 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2022-02-21T16:59:15.178 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2022-02-21T16:59:15.178 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2022-02-21T16:59:15.178 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2022-02-21T16:59:15.179 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-02-21T16:59:15.179 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-02-21T16:59:15.179 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-02-21T16:59:15.179 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-02-21T16:59:15.180 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2022-02-21T16:59:15.180 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2022-02-21T16:59:15.224 INFO:teuthology.orchestra.run.smithi033.stdout:done 2022-02-21T16:59:15.254 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2022-02-21T16:59:15.275 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T16:59:15.305 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T16:59:15.305 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2022-02-21T16:59:15.305 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2022-02-21T16:59:15.305 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2022-02-21T16:59:15.306 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 19 2022-02-21T16:59:15.306 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T16:59:15.306 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2022-02-21T16:59:15.306 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta-hwe-5.13 2022-02-21T16:59:15.307 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T16:59:15.307 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T16:59:15.307 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T16:59:15.307 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2022-02-21T16:59:15.307 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2022-02-21T16:59:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2022-02-21T16:59:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2022-02-21T16:59:15.308 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-02-21T16:59:15.308 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-02-21T16:59:15.309 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-02-21T16:59:15.309 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-02-21T16:59:15.309 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2022-02-21T16:59:15.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2022-02-21T16:59:15.396 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T16:59:15.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2022-02-21T16:59:15.512 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T16:59:15.598 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T16:59:15.600 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T16:59:15.634 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T16:59:15.634 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2022-02-21T16:59:15.635 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2022-02-21T16:59:15.635 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2022-02-21T16:59:15.635 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 19 2022-02-21T16:59:15.635 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T16:59:15.635 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2022-02-21T16:59:15.636 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta-hwe-5.13 2022-02-21T16:59:15.636 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T16:59:15.636 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T16:59:15.636 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T16:59:15.637 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2022-02-21T16:59:15.637 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2022-02-21T16:59:15.637 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2022-02-21T16:59:15.637 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2022-02-21T16:59:15.637 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-02-21T16:59:15.638 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-02-21T16:59:15.638 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-02-21T16:59:15.638 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-02-21T16:59:15.638 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2022-02-21T16:59:15.639 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2022-02-21T16:59:15.667 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T16:59:15.689 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T16:59:15.696 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2022-02-21T16:59:15.744 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T16:59:15.834 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T16:59:15.849 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T16:59:15.965 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T16:59:16.056 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T16:59:16.120 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T16:59:16.146 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T16:59:16.987 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-02-21T16:59:17.010 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T16:59:17.078 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-02-21T16:59:17.240 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2022-02-21T16:59:17.255 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2022-02-21T16:59:17.256 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2022-02-21T16:59:17.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T16:59:17.334 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2022-02-21T16:59:17.411 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-02-21T16:59:17.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-02-21T16:59:17.443 INFO:teuthology.orchestra.run.smithi005.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-02-21T16:59:17.443 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T16:59:17.445 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T16:59:17.465 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T16:59:17.465 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2022-02-21T16:59:17.466 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2022-02-21T16:59:17.466 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2022-02-21T16:59:17.466 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 19 2022-02-21T16:59:17.466 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T16:59:17.467 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2022-02-21T16:59:17.467 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta-hwe-5.13 2022-02-21T16:59:17.467 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T16:59:17.467 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T16:59:17.468 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T16:59:17.468 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2022-02-21T16:59:17.468 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2022-02-21T16:59:17.468 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2022-02-21T16:59:17.468 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2022-02-21T16:59:17.469 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2022-02-21T16:59:17.508 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-02-21T16:59:17.513 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2022-02-21T16:59:17.513 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2022-02-21T16:59:17.515 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.HD7JTEv7F4 2022-02-21T16:59:17.515 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub/grub.cfg /tmp/tmp.HD7JTEv7F4 2022-02-21T16:59:17.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.HD7JTEv7F4 2022-02-21T16:59:17.598 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.HD7JTEv7F4 is 8KB 2022-02-21T16:59:17.611 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.HD7JTEv7F4 2022-02-21T16:59:17.618 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-02-21T16:59:17.676 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T16:59:17.676 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-02-21T16:59:17.677 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-02-21T16:59:17.684 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2022-02-21T16:59:17.685 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2022-02-21T16:59:17.702 INFO:teuthology.orchestra.run.smithi067.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-02-21T16:59:17.703 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T16:59:17.704 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T16:59:17.705 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-02-21T16:59:17.706 DEBUG:teuthology.orchestra.run.smithi005:> sudo update-grub 2022-02-21T16:59:17.726 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T16:59:17.726 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2022-02-21T16:59:17.726 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2022-02-21T16:59:17.727 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2022-02-21T16:59:17.727 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 19 2022-02-21T16:59:17.727 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T16:59:17.727 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2022-02-21T16:59:17.728 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta-hwe-5.13 2022-02-21T16:59:17.728 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T16:59:17.728 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T16:59:17.728 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T16:59:17.728 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2022-02-21T16:59:17.729 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2022-02-21T16:59:17.729 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2022-02-21T16:59:17.729 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2022-02-21T16:59:17.729 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2022-02-21T16:59:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.DbumaKJXI1 2022-02-21T16:59:17.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub/grub.cfg /tmp/tmp.DbumaKJXI1 2022-02-21T16:59:17.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.DbumaKJXI1 2022-02-21T16:59:17.826 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.DbumaKJXI1 is 8KB 2022-02-21T16:59:17.841 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.DbumaKJXI1 2022-02-21T16:59:17.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-02-21T16:59:17.874 INFO:teuthology.orchestra.run.smithi033.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-02-21T16:59:17.874 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded. 2022-02-21T16:59:17.876 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic-hwe-20.04 2022-02-21T16:59:17.893 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic-hwe-20.04 2022-02-21T16:59:17.894 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2022-02-21T16:59:17.894 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2022-02-21T16:59:17.894 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2022-02-21T16:59:17.894 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 19 2022-02-21T16:59:17.895 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2022-02-21T16:59:17.895 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2022-02-21T16:59:17.895 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta-hwe-5.13 2022-02-21T16:59:17.895 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.13.0.30.33~20.04.17 2022-02-21T16:59:17.895 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-02-21T16:59:17.896 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-02-21T16:59:17.896 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2022-02-21T16:59:17.896 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2022-02-21T16:59:17.896 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2022-02-21T16:59:17.897 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2022-02-21T16:59:17.897 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2022-02-21T16:59:17.902 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.ZHG7C3UMGb 2022-02-21T16:59:17.903 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ZHG7C3UMGb 2022-02-21T16:59:17.906 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T16:59:17.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-02-21T16:59:17.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-02-21T16:59:17.943 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-02-21T16:59:17.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo update-grub 2022-02-21T16:59:17.960 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.ZHG7C3UMGb 2022-02-21T16:59:17.995 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.ZHG7C3UMGb is 8KB 2022-02-21T16:59:18.008 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.ZHG7C3UMGb 2022-02-21T16:59:18.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-02-21T16:59:18.071 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T16:59:18.072 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-02-21T16:59:18.072 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-02-21T16:59:18.105 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-02-21T16:59:18.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo update-grub 2022-02-21T16:59:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub' 2022-02-21T16:59:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T16:59:18.970 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2022-02-21T16:59:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub' 2022-02-21T16:59:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T16:59:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2022-02-21T16:59:19.234 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T16:59:19.246 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T16:59:19.369 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub' 2022-02-21T16:59:19.372 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-02-21T16:59:19.380 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2022-02-21T16:59:19.492 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T16:59:19.504 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T16:59:19.602 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T16:59:19.614 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T16:59:19.651 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-02-21T16:59:19.663 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-02-21T16:59:19.836 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T16:59:19.848 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T16:59:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-02-21T16:59:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-02-21T16:59:21.210 INFO:teuthology.orchestra.run.smithi005.stderr:done 2022-02-21T16:59:21.211 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2022-02-21T16:59:21.240 INFO:teuthology.orchestra.run.smithi067.stderr:done 2022-02-21T16:59:21.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2022-02-21T16:59:21.661 INFO:teuthology.orchestra.run.smithi033.stderr:done 2022-02-21T16:59:21.662 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2022-02-21T16:59:51.242 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-02-21T16:59:51.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2022-02-21T16:59:51.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T16:59:51.246 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-02-21T16:59:51.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2022-02-21T16:59:51.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T16:59:51.666 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-02-21T16:59:51.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2022-02-21T16:59:51.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T17:00:09.803 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2022-02-21T17:00:09.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2022-02-21T17:00:10.058 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2022-02-21T17:00:39.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2022-02-21T17:00:39.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T17:00:39.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2022-02-21T17:00:39.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T17:00:40.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2022-02-21T17:00:40.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T17:00:55.305 DEBUG:teuthology.orchestra.run.smithi005:> true 2022-02-21T17:00:55.809 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2022-02-21T17:00:55.810 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-30-generic"... 2022-02-21T17:00:55.810 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2022-02-21T17:00:55.815 INFO:teuthology.orchestra.run.smithi005.stdout:5.13.0-30-generic 2022-02-21T17:00:55.816 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-02-21T17:00:55.816 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-02-21T17:00:55.816 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-02-21T17:00:55.972 DEBUG:teuthology.orchestra.run.smithi033:> true 2022-02-21T17:00:56.723 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2022-02-21T17:00:56.724 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-30-generic"... 2022-02-21T17:00:56.724 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2022-02-21T17:00:56.729 INFO:teuthology.orchestra.run.smithi033.stdout:5.13.0-30-generic 2022-02-21T17:00:56.730 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-02-21T17:00:56.730 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-02-21T17:00:56.730 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-02-21T17:00:56.818 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-02-21T17:00:56.818 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-02-21T17:00:56.885 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2022-02-21T17:00:56.912 DEBUG:teuthology.parallel:result is None 2022-02-21T17:00:57.732 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-02-21T17:00:57.732 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-02-21T17:00:57.871 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2022-02-21T17:00:57.893 DEBUG:teuthology.parallel:result is None 2022-02-21T17:00:58.187 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2022-02-21T17:01:28.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2022-02-21T17:01:28.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T17:01:28.377 DEBUG:teuthology.orchestra.run.smithi067:> true 2022-02-21T17:01:28.923 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2022-02-21T17:01:28.923 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-30-generic"... 2022-02-21T17:01:28.924 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2022-02-21T17:01:28.929 INFO:teuthology.orchestra.run.smithi067.stdout:5.13.0-30-generic 2022-02-21T17:01:28.930 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-02-21T17:01:28.930 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-02-21T17:01:28.930 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-02-21T17:01:29.931 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-02-21T17:01:29.932 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-02-21T17:01:30.012 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2022-02-21T17:01:30.031 DEBUG:teuthology.parallel:result is None 2022-02-21T17:01:30.032 INFO:teuthology.run_tasks:Running task internal.base... 2022-02-21T17:01:30.043 INFO:teuthology.task.internal:Creating test directory... 2022-02-21T17:01:30.043 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-02-21T17:01:30.046 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-02-21T17:01:30.049 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-02-21T17:01:30.056 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-02-21T17:01:30.090 INFO:teuthology.run_tasks:Running task internal.archive... 2022-02-21T17:01:30.109 INFO:teuthology.task.internal:Creating archive directory... 2022-02-21T17:01:30.110 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-02-21T17:01:30.112 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-02-21T17:01:30.115 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-02-21T17:01:30.137 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-02-21T17:01:30.147 INFO:teuthology.task.internal:Enabling coredump saving... 2022-02-21T17:01:30.148 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-02-21T17:01:30.150 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-02-21T17:01:30.153 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-02-21T17:01:30.172 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T17:01:30.176 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T17:01:30.178 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T17:01:30.179 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T17:01:30.186 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T17:01:30.188 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-02-21T17:01:30.189 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-02-21T17:01:30.201 INFO:teuthology.task.internal:Configuring sudo... 2022-02-21T17:01:30.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-02-21T17:01:30.235 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-02-21T17:01:30.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-02-21T17:01:30.253 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-02-21T17:01:30.266 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-02-21T17:01:30.267 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-02-21T17:01:30.270 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-02-21T17:01:30.296 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-02-21T17:01:30.303 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-02-21T17:01:30.311 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-02-21T17:01:30.360 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:01:30.360 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-02-21T17:01:30.378 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-02-21T17:01:30.386 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-02-21T17:01:30.392 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:01:30.393 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-02-21T17:01:30.459 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-02-21T17:01:30.467 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-02-21T17:01:30.516 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:01:30.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-02-21T17:01:30.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2022-02-21T17:01:30.546 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2022-02-21T17:01:30.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2022-02-21T17:01:30.597 INFO:teuthology.run_tasks:Running task internal.timer... 2022-02-21T17:01:30.605 INFO:teuthology.task.internal:Starting timer... 2022-02-21T17:01:30.605 INFO:teuthology.run_tasks:Running task pcp... 2022-02-21T17:01:30.653 INFO:teuthology.run_tasks:Running task selinux... 2022-02-21T17:01:30.679 DEBUG:teuthology.task.selinux:Excluding smithi005: OS 'ubuntu' does not support SELinux 2022-02-21T17:01:30.680 DEBUG:teuthology.task.selinux:Excluding smithi033: OS 'ubuntu' does not support SELinux 2022-02-21T17:01:30.680 DEBUG:teuthology.task.selinux:Excluding smithi067: OS 'ubuntu' does not support SELinux 2022-02-21T17:01:30.681 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-02-21T17:01:30.681 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-02-21T17:01:30.681 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-02-21T17:01:30.682 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-02-21T17:01:30.726 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-02-21T17:01:30.732 INFO:teuthology.repo_utils:Fetching master from origin 2022-02-21T17:01:30.765 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-02-21T17:01:30.786 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-02-21T17:01:30.787 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi033.front.sepia.ceph.com,smithi067.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-02-21T17:04:31.494 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi067.front.sepia.ceph.com')] 2022-02-21T17:04:31.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2022-02-21T17:04:31.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T17:04:31.553 DEBUG:teuthology.orchestra.run.smithi005:> true 2022-02-21T17:04:31.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2022-02-21T17:04:31.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T17:04:31.723 DEBUG:teuthology.orchestra.run.smithi033:> true 2022-02-21T17:04:31.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2022-02-21T17:04:31.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-02-21T17:04:31.895 DEBUG:teuthology.orchestra.run.smithi067:> true 2022-02-21T17:04:32.021 INFO:teuthology.run_tasks:Running task clock... 2022-02-21T17:04:32.039 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-02-21T17:04:32.040 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-02-21T17:04:32.043 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-02-21T17:04:32.046 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-02-21T17:04:32.047 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-02-21T17:04:32.050 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-02-21T17:04:32.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-02-21T17:04:32.080 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-02-21T17:04:32.081 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: Command line: ntpd -gq 2022-02-21T17:04:32.081 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: proto: precision = 0.103 usec (-23) 2022-02-21T17:04:32.082 INFO:teuthology.orchestra.run.smithi005.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T17:04:32.082 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T17:04:32.083 INFO:teuthology.orchestra.run.smithi005.stderr:restrict ::: KOD does nothing without LIMITED. 2022-02-21T17:04:32.084 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: restrict ::: KOD does nothing without LIMITED. 2022-02-21T17:04:32.084 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: Listen and drop on 0 v6wildcard [::]:123 2022-02-21T17:04:32.085 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-02-21T17:04:32.085 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: Listen normally on 2 lo 127.0.0.1:123 2022-02-21T17:04:32.085 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: Listen normally on 3 ens1f0 172.21.15.5:123 2022-02-21T17:04:32.086 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: Listen normally on 4 lo [::1]:123 2022-02-21T17:04:32.086 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1442%4]:123 2022-02-21T17:04:32.086 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:32 ntpd[8815]: Listening on routing socket on fd #22 for interface updates 2022-02-21T17:04:32.087 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-02-21T17:04:32.087 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: Command line: ntpd -gq 2022-02-21T17:04:32.087 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: proto: precision = 0.060 usec (-24) 2022-02-21T17:04:32.088 INFO:teuthology.orchestra.run.smithi033.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T17:04:32.088 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T17:04:32.088 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: restrict ::: KOD does nothing without LIMITED. 2022-02-21T17:04:32.089 INFO:teuthology.orchestra.run.smithi033.stderr:restrict ::: KOD does nothing without LIMITED. 2022-02-21T17:04:32.091 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: Listen and drop on 0 v6wildcard [::]:123 2022-02-21T17:04:32.091 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-02-21T17:04:32.091 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: Listen normally on 2 lo 127.0.0.1:123 2022-02-21T17:04:32.091 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: Listen normally on 3 ens1f0 172.21.15.33:123 2022-02-21T17:04:32.092 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: Listen normally on 4 lo [::1]:123 2022-02-21T17:04:32.092 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:d294%4]:123 2022-02-21T17:04:32.092 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:32 ntpd[8830]: Listening on routing socket on fd #22 for interface updates 2022-02-21T17:04:32.093 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-02-21T17:04:32.093 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: Command line: ntpd -gq 2022-02-21T17:04:32.093 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: proto: precision = 0.080 usec (-23) 2022-02-21T17:04:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T17:04:32.095 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-02-21T17:04:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:restrict ::: KOD does nothing without LIMITED. 2022-02-21T17:04:32.096 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: restrict ::: KOD does nothing without LIMITED. 2022-02-21T17:04:32.097 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: Listen and drop on 0 v6wildcard [::]:123 2022-02-21T17:04:32.097 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-02-21T17:04:32.097 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: Listen normally on 2 lo 127.0.0.1:123 2022-02-21T17:04:32.097 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: Listen normally on 3 enp3s0f1 172.21.15.67:123 2022-02-21T17:04:32.098 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: Listen normally on 4 lo [::1]:123 2022-02-21T17:04:32.098 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:fc0b%5]:123 2022-02-21T17:04:32.098 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:32 ntpd[8922]: Listening on routing socket on fd #22 for interface updates 2022-02-21T17:04:33.081 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:33.081 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:33.081 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:33.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:33.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:33.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:33.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:33.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:33.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:33.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:33.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:33.085 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:33 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:33.087 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:33.088 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:33.088 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:33.088 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:33.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:33.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:33.090 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:33.090 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:33.090 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:33.090 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:33.091 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:33 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:34.081 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:34 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:34.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:34 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:34.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:34 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:34.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:34 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:34.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:34 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:34.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:34 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:34 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:34 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:34.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:34 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:34.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:34 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:34.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:34 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:34.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:34 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:35.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:35.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:35.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:35.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:35 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.088 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:35.089 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:35 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:35.093 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:35.093 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:35.093 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:35.093 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:35 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:36.081 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:36 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:36.081 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:36 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:36.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:36 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:36.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:36 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:36.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:36 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:36.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:36 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:36.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:36.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:36.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:36.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:36.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:36.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:36.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:36.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:36.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:36.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:36.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:36.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:36 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:36 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:36 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:36 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:36 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:36 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:36 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:37.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.085 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:37.085 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:37.085 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.085 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.086 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:37.086 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:37.086 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.086 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.086 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:37.087 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:37 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:37.090 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.090 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.090 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.091 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.091 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.091 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.091 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:37.092 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:37.092 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.092 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.092 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:37.093 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:37 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:37.093 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.094 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.094 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.094 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.094 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.095 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.095 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.095 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:37 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:38.081 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:38 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:38.081 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:38 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:38.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:38 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:38.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:38 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:38.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:38 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:38.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:38 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:38.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:38.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:38.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:38.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:38.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:38.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:38.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:38.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:38.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:38.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:38.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:38.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:38 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:38 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:38 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:38 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:38 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:38 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:38 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:39.081 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.082 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.083 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:39.084 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:39.084 INFO:teuthology.orchestra.run.smithi005.stdout:21 Feb 17:04:39 ntpd[8815]: ntpd: time slew -0.000334 s 2022-02-21T17:04:39.085 INFO:teuthology.orchestra.run.smithi005.stdout:ntpd: time slew -0.000334s 2022-02-21T17:04:39.085 INFO:teuthology.orchestra.run.smithi005.stderr:21 Feb 17:04:39 ntpd[8815]: can't open /var/log/ntpstats/loopstats.20220221: Permission denied 2022-02-21T17:04:39.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.088 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.088 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.088 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.088 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.088 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.089 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:39.089 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:39.089 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.089 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.090 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:39.090 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:39 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:39.091 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.091 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:39.093 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:39.094 INFO:teuthology.orchestra.run.smithi067.stdout:21 Feb 17:04:39 ntpd[8922]: ntpd: time slew -0.000003 s 2022-02-21T17:04:39.094 INFO:teuthology.orchestra.run.smithi067.stdout:ntpd: time slew -0.000003s 2022-02-21T17:04:39.095 INFO:teuthology.orchestra.run.smithi067.stderr:21 Feb 17:04:39 ntpd[8922]: can't open /var/log/ntpstats/loopstats.20220221: Permission denied 2022-02-21T17:04:39.136 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2022-02-21T17:04:39.136 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2022-02-21T17:04:39.136 INFO:teuthology.orchestra.run.smithi005.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:39.137 INFO:teuthology.orchestra.run.smithi005.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:39.137 INFO:teuthology.orchestra.run.smithi005.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:39.137 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:39.150 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2022-02-21T17:04:39.150 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2022-02-21T17:04:39.150 INFO:teuthology.orchestra.run.smithi067.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:39.151 INFO:teuthology.orchestra.run.smithi067.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:39.151 INFO:teuthology.orchestra.run.smithi067.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:39.151 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:40.084 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:40 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:40.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:40 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:40.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:40 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:40.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:40 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:40.085 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:40 ntpd[8830]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-02-21T17:04:40.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:40 ntpd[8830]: can't open /var/log/ntpstats/rawstats.20220221: Permission denied 2022-02-21T17:04:40.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:40 ntpd[8830]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-02-21T17:04:40.086 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:40 ntpd[8830]: can't open /var/log/ntpstats/peerstats.20220221: Permission denied 2022-02-21T17:04:40.086 INFO:teuthology.orchestra.run.smithi033.stdout:21 Feb 17:04:40 ntpd[8830]: ntpd: time slew -0.000155 s 2022-02-21T17:04:40.087 INFO:teuthology.orchestra.run.smithi033.stdout:ntpd: time slew -0.000155s 2022-02-21T17:04:40.087 INFO:teuthology.orchestra.run.smithi033.stderr:21 Feb 17:04:40 ntpd[8830]: can't open /var/log/ntpstats/loopstats.20220221: Permission denied 2022-02-21T17:04:40.132 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2022-02-21T17:04:40.132 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2022-02-21T17:04:40.133 INFO:teuthology.orchestra.run.smithi033.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:40.133 INFO:teuthology.orchestra.run.smithi033.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:40.133 INFO:teuthology.orchestra.run.smithi033.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:40.133 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-02-21T17:04:40.134 INFO:teuthology.run_tasks:Running task kubeadm... 2022-02-21T17:04:40.153 INFO:tasks.kubeadm:Kubeadm start 2022-02-21T17:04:40.154 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-02-21T17:04:40.155 DEBUG:teuthology.orchestra.run.smithi005:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-02-21T17:04:40.157 DEBUG:teuthology.orchestra.run.smithi033:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-02-21T17:04:40.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-02-21T17:04:40.183 INFO:teuthology.orchestra.run.smithi005.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-02-21T17:04:40.192 INFO:teuthology.orchestra.run.smithi005.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-02-21T17:04:40.201 INFO:teuthology.orchestra.run.smithi033.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-02-21T17:04:40.202 INFO:teuthology.orchestra.run.smithi005.stdout:net.ipv4.ip_forward = 1 2022-02-21T17:04:40.205 INFO:teuthology.orchestra.run.smithi067.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-02-21T17:04:40.210 INFO:teuthology.orchestra.run.smithi033.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-02-21T17:04:40.214 INFO:teuthology.orchestra.run.smithi067.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-02-21T17:04:40.217 INFO:teuthology.orchestra.run.smithi033.stdout:net.ipv4.ip_forward = 1 2022-02-21T17:04:40.221 INFO:teuthology.orchestra.run.smithi067.stdout:net.ipv4.ip_forward = 1 2022-02-21T17:04:40.290 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:04:40.291 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-02-21T17:04:40.307 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:04:40.307 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/docker/daemon.json 2022-02-21T17:04:40.366 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:04:40.367 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-02-21T17:04:40.383 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:04:40.384 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/docker/daemon.json 2022-02-21T17:04:40.440 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:04:40.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-02-21T17:04:40.457 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:04:40.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/docker/daemon.json 2022-02-21T17:04:40.513 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl restart docker || true 2022-02-21T17:04:40.516 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl restart docker || true 2022-02-21T17:04:40.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl restart docker || true 2022-02-21T17:04:41.781 INFO:tasks.kubeadm:Identifying pod subnet 2022-02-21T17:04:41.788 INFO:tasks.kubeadm:172.21.15.5 is in 172.21.0.0/20 at pos 3844 2022-02-21T17:04:41.802 INFO:tasks.kubeadm:Pod subnet: 10.248.32.0/21 2022-02-21T17:04:41.803 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-02-21T17:04:41.803 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-02-21T17:04:41.806 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-02-21T17:04:41.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-02-21T17:04:41.826 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-02-21T17:04:41.826 INFO:teuthology.orchestra.run.smithi005.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:41.827 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-02-21T17:04:41.836 INFO:teuthology.orchestra.run.smithi033.stderr: 2022-02-21T17:04:41.837 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:41.837 INFO:teuthology.orchestra.run.smithi033.stderr: 2022-02-21T17:04:41.837 INFO:teuthology.orchestra.run.smithi067.stderr: 2022-02-21T17:04:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:41.838 INFO:teuthology.orchestra.run.smithi067.stderr: 2022-02-21T17:04:41.952 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T17:04:41.957 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T17:04:42.071 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T17:04:42.071 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T17:04:42.153 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T17:04:42.238 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T17:04:42.454 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T17:04:42.518 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T17:04:42.542 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T17:04:42.605 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T17:04:42.634 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T17:04:42.697 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T17:04:43.465 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2022-02-21T17:04:43.643 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2022-02-21T17:04:43.644 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2022-02-21T17:04:43.742 INFO:teuthology.orchestra.run.smithi067.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T17:04:43.755 INFO:teuthology.orchestra.run.smithi067.stderr: 2022-02-21T17:04:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:43.756 INFO:teuthology.orchestra.run.smithi067.stderr: 2022-02-21T17:04:43.812 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2022-02-21T17:04:43.841 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-02-21T17:04:43.902 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-02-21T17:04:43.974 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2022-02-21T17:04:43.974 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2022-02-21T17:04:44.017 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2022-02-21T17:04:44.018 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2022-02-21T17:04:44.082 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2022-02-21T17:04:44.083 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2022-02-21T17:04:44.109 INFO:teuthology.orchestra.run.smithi033.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T17:04:44.127 INFO:teuthology.orchestra.run.smithi033.stderr: 2022-02-21T17:04:44.127 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:44.128 INFO:teuthology.orchestra.run.smithi033.stderr: 2022-02-21T17:04:44.144 INFO:teuthology.orchestra.run.smithi067.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-02-21T17:04:44.145 INFO:teuthology.orchestra.run.smithi067.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-02-21T17:04:44.145 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2022-02-21T17:04:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: apt-transport-https 2022-02-21T17:04:44.176 INFO:teuthology.orchestra.run.smithi005.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T17:04:44.182 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-02-21T17:04:44.189 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-02-21T17:04:44.189 INFO:teuthology.orchestra.run.smithi005.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:44.189 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-02-21T17:04:44.246 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-02-21T17:04:44.331 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 1 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T17:04:44.332 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 4,680 B of archives. 2022-02-21T17:04:44.332 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 162 kB of additional disk space will be used. 2022-02-21T17:04:44.332 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-02-21T17:04:44.363 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2022-02-21T17:04:44.363 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2022-02-21T17:04:44.423 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2022-02-21T17:04:44.424 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2022-02-21T17:04:44.557 INFO:teuthology.orchestra.run.smithi033.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-02-21T17:04:44.558 INFO:teuthology.orchestra.run.smithi033.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-02-21T17:04:44.558 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2022-02-21T17:04:44.559 INFO:teuthology.orchestra.run.smithi033.stdout: apt-transport-https 2022-02-21T17:04:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T17:04:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T17:04:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: falling back to frontend: Readline 2022-02-21T17:04:44.585 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T17:04:44.585 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T17:04:44.585 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: falling back to frontend: Teletype 2022-02-21T17:04:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T17:04:44.612 INFO:teuthology.orchestra.run.smithi005.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-02-21T17:04:44.613 INFO:teuthology.orchestra.run.smithi005.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-02-21T17:04:44.613 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2022-02-21T17:04:44.614 INFO:teuthology.orchestra.run.smithi005.stdout: apt-transport-https 2022-02-21T17:04:44.641 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 4,680 B in 0s (26.1 kB/s) 2022-02-21T17:04:44.712 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package apt-transport-https. 2022-02-21T17:04:44.747 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 1 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T17:04:44.747 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 4,680 B of archives. 2022-02-21T17:04:44.747 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 162 kB of additional disk space will be used. 2022-02-21T17:04:44.748 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-02-21T17:04:44.757 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-02-21T17:04:44.760 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-02-21T17:04:44.800 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 1 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T17:04:44.801 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 4,680 B of archives. 2022-02-21T17:04:44.801 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 162 kB of additional disk space will be used. 2022-02-21T17:04:44.801 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-02-21T17:04:44.802 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-02-21T17:04:44.988 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T17:04:44.988 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T17:04:44.988 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: falling back to frontend: Readline 2022-02-21T17:04:45.005 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T17:04:45.005 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T17:04:45.005 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: falling back to frontend: Teletype 2022-02-21T17:04:45.011 INFO:teuthology.orchestra.run.smithi033.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T17:04:45.027 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T17:04:45.028 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T17:04:45.028 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: falling back to frontend: Readline 2022-02-21T17:04:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T17:04:45.053 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T17:04:45.053 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: falling back to frontend: Teletype 2022-02-21T17:04:45.057 INFO:teuthology.orchestra.run.smithi005.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T17:04:45.062 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 4,680 B in 0s (26.2 kB/s) 2022-02-21T17:04:45.081 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up apt-transport-https (2.0.6) ... 2022-02-21T17:04:45.107 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 4,680 B in 0s (26.2 kB/s) 2022-02-21T17:04:45.130 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package apt-transport-https. 2022-02-21T17:04:45.168 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-02-21T17:04:45.171 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-02-21T17:04:45.195 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package apt-transport-https. 2022-02-21T17:04:45.205 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-02-21T17:04:45.236 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-02-21T17:04:45.239 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-02-21T17:04:45.277 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-02-21T17:04:45.475 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up apt-transport-https (2.0.6) ... 2022-02-21T17:04:45.562 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up apt-transport-https (2.0.6) ... 2022-02-21T17:04:45.597 INFO:teuthology.orchestra.run.smithi067.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-02-21T17:04:45.613 INFO:teuthology.orchestra.run.smithi067.stderr: 2022-02-21T17:04:45.614 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:45.614 INFO:teuthology.orchestra.run.smithi067.stderr: 2022-02-21T17:04:45.933 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T17:04:45.934 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T17:04:46.021 INFO:teuthology.orchestra.run.smithi033.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-02-21T17:04:46.030 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T17:04:46.038 INFO:teuthology.orchestra.run.smithi033.stderr: 2022-02-21T17:04:46.038 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:46.038 INFO:teuthology.orchestra.run.smithi033.stderr: 2022-02-21T17:04:46.140 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T17:04:46.157 INFO:teuthology.orchestra.run.smithi005.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-02-21T17:04:46.169 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-02-21T17:04:46.169 INFO:teuthology.orchestra.run.smithi005.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:46.169 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-02-21T17:04:46.224 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T17:04:46.330 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-02-21T17:04:46.333 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T17:04:46.439 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T17:04:46.474 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-02-21T17:04:46.474 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-02-21T17:04:46.542 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T17:04:46.564 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-02-21T17:04:46.624 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-02-21T17:04:46.655 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-02-21T17:04:46.661 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-02-21T17:04:46.779 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 63.3 kB in 1s (62.9 kB/s) 2022-02-21T17:04:46.795 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-02-21T17:04:46.920 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-02-21T17:04:47.076 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 63.3 kB in 1s (71.4 kB/s) 2022-02-21T17:04:47.147 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-02-21T17:04:47.284 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 63.3 kB in 1s (65.7 kB/s) 2022-02-21T17:04:47.646 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2022-02-21T17:04:47.824 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2022-02-21T17:04:47.825 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2022-02-21T17:04:47.920 INFO:teuthology.orchestra.run.smithi067.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T17:04:47.938 INFO:teuthology.orchestra.run.smithi067.stderr: 2022-02-21T17:04:47.938 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:47.939 INFO:teuthology.orchestra.run.smithi067.stderr: 2022-02-21T17:04:47.950 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-02-21T17:04:47.995 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2022-02-21T17:04:48.128 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2022-02-21T17:04:48.128 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2022-02-21T17:04:48.163 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-02-21T17:04:48.173 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2022-02-21T17:04:48.174 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2022-02-21T17:04:48.219 INFO:teuthology.orchestra.run.smithi033.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T17:04:48.232 INFO:teuthology.orchestra.run.smithi033.stderr: 2022-02-21T17:04:48.232 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:48.232 INFO:teuthology.orchestra.run.smithi033.stderr: 2022-02-21T17:04:48.287 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-02-21T17:04:48.330 INFO:teuthology.orchestra.run.smithi067.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-02-21T17:04:48.331 INFO:teuthology.orchestra.run.smithi067.stdout:bridge-utils set to manually installed. 2022-02-21T17:04:48.331 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2022-02-21T17:04:48.331 INFO:teuthology.orchestra.run.smithi067.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-02-21T17:04:48.332 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2022-02-21T17:04:48.332 INFO:teuthology.orchestra.run.smithi067.stdout: nftables 2022-02-21T17:04:48.344 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2022-02-21T17:04:48.345 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2022-02-21T17:04:48.383 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2022-02-21T17:04:48.384 INFO:teuthology.orchestra.run.smithi067.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-02-21T17:04:48.435 INFO:teuthology.orchestra.run.smithi005.stdout:48 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-02-21T17:04:48.448 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-02-21T17:04:48.448 INFO:teuthology.orchestra.run.smithi005.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-02-21T17:04:48.449 INFO:teuthology.orchestra.run.smithi005.stderr: 2022-02-21T17:04:48.465 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2022-02-21T17:04:48.466 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2022-02-21T17:04:48.503 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2022-02-21T17:04:48.580 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 8 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T17:04:48.580 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 73.6 MB of archives. 2022-02-21T17:04:48.580 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 321 MB of additional disk space will be used. 2022-02-21T17:04:48.581 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-02-21T17:04:48.610 INFO:teuthology.orchestra.run.smithi033.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-02-21T17:04:48.610 INFO:teuthology.orchestra.run.smithi033.stdout:bridge-utils set to manually installed. 2022-02-21T17:04:48.610 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2022-02-21T17:04:48.611 INFO:teuthology.orchestra.run.smithi033.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-02-21T17:04:48.611 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2022-02-21T17:04:48.612 INFO:teuthology.orchestra.run.smithi033.stdout: nftables 2022-02-21T17:04:48.623 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-02-21T17:04:48.658 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2022-02-21T17:04:48.660 INFO:teuthology.orchestra.run.smithi033.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-02-21T17:04:48.682 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2022-02-21T17:04:48.682 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2022-02-21T17:04:48.837 INFO:teuthology.orchestra.run.smithi005.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-02-21T17:04:48.838 INFO:teuthology.orchestra.run.smithi005.stdout:bridge-utils set to manually installed. 2022-02-21T17:04:48.838 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2022-02-21T17:04:48.839 INFO:teuthology.orchestra.run.smithi005.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-02-21T17:04:48.839 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2022-02-21T17:04:48.840 INFO:teuthology.orchestra.run.smithi005.stdout: nftables 2022-02-21T17:04:48.865 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 8 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T17:04:48.866 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 73.6 MB of archives. 2022-02-21T17:04:48.866 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 321 MB of additional disk space will be used. 2022-02-21T17:04:48.866 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-02-21T17:04:48.889 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2022-02-21T17:04:48.890 INFO:teuthology.orchestra.run.smithi005.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-02-21T17:04:48.919 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-02-21T17:04:48.924 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-02-21T17:04:48.965 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-02-21T17:04:49.047 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-02-21T17:04:49.104 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 8 newly installed, 0 to remove and 48 not upgraded. 2022-02-21T17:04:49.104 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 73.6 MB of archives. 2022-02-21T17:04:49.104 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 321 MB of additional disk space will be used. 2022-02-21T17:04:49.104 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-02-21T17:04:49.188 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-02-21T17:04:49.194 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-02-21T17:04:49.248 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-02-21T17:04:49.324 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-02-21T17:04:49.392 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-02-21T17:04:49.564 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-02-21T17:04:49.576 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-02-21T17:04:50.026 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-02-21T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-02-21T17:04:50.282 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-02-21T17:04:50.400 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-02-21T17:04:50.715 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-02-21T17:04:50.799 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T17:04:50.799 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T17:04:50.799 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: falling back to frontend: Readline 2022-02-21T17:04:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T17:04:50.807 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T17:04:50.807 INFO:teuthology.orchestra.run.smithi067.stderr:debconf: falling back to frontend: Teletype 2022-02-21T17:04:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T17:04:50.866 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 73.6 MB in 2s (33.8 MB/s) 2022-02-21T17:04:50.915 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-02-21T17:04:50.943 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package conntrack. 2022-02-21T17:04:50.986 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-02-21T17:04:50.990 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-02-21T17:04:51.025 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-02-21T17:04:51.046 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-02-21T17:04:51.048 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-02-21T17:04:51.204 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-02-21T17:04:51.294 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cri-tools. 2022-02-21T17:04:51.309 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-02-21T17:04:51.361 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-02-21T17:04:51.472 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T17:04:51.472 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T17:04:51.472 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: falling back to frontend: Readline 2022-02-21T17:04:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T17:04:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T17:04:51.480 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: falling back to frontend: Teletype 2022-02-21T17:04:51.486 INFO:teuthology.orchestra.run.smithi033.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T17:04:51.529 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: unable to initialize frontend: Dialog 2022-02-21T17:04:51.529 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-02-21T17:04:51.530 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: falling back to frontend: Readline 2022-02-21T17:04:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: unable to initialize frontend: Readline 2022-02-21T17:04:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: (This frontend requires a controlling tty.) 2022-02-21T17:04:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:debconf: falling back to frontend: Teletype 2022-02-21T17:04:51.541 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 73.6 MB in 3s (28.6 MB/s) 2022-02-21T17:04:51.544 INFO:teuthology.orchestra.run.smithi005.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-02-21T17:04:51.595 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 73.6 MB in 2s (30.5 MB/s) 2022-02-21T17:04:51.629 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package conntrack. 2022-02-21T17:04:51.674 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-02-21T17:04:51.677 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-02-21T17:04:51.684 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package conntrack. 2022-02-21T17:04:51.727 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-02-21T17:04:51.728 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-02-21T17:04:51.731 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-02-21T17:04:51.890 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-02-21T17:04:52.140 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package cri-tools. 2022-02-21T17:04:52.154 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-02-21T17:04:52.177 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package cri-tools. 2022-02-21T17:04:52.190 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-02-21T17:04:52.199 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-02-21T17:04:52.218 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-02-21T17:04:52.481 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ebtables. 2022-02-21T17:04:52.494 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-02-21T17:04:52.547 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-02-21T17:04:52.817 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package kubernetes-cni. 2022-02-21T17:04:52.831 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-02-21T17:04:52.875 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-02-21T17:04:53.279 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ebtables. 2022-02-21T17:04:53.293 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-02-21T17:04:53.343 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ebtables. 2022-02-21T17:04:53.345 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-02-21T17:04:53.351 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-02-21T17:04:53.409 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-02-21T17:04:53.599 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package kubernetes-cni. 2022-02-21T17:04:53.612 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-02-21T17:04:53.657 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-02-21T17:04:53.696 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package kubernetes-cni. 2022-02-21T17:04:53.709 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-02-21T17:04:53.754 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-02-21T17:04:54.302 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package socat. 2022-02-21T17:04:54.315 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-02-21T17:04:54.352 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking socat (1.7.3.3-2) ... 2022-02-21T17:04:54.739 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package kubelet. 2022-02-21T17:04:54.752 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-02-21T17:04:54.789 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking kubelet (1.23.4-00) ... 2022-02-21T17:04:55.068 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package socat. 2022-02-21T17:04:55.082 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-02-21T17:04:55.126 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking socat (1.7.3.3-2) ... 2022-02-21T17:04:55.223 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package socat. 2022-02-21T17:04:55.236 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-02-21T17:04:55.282 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking socat (1.7.3.3-2) ... 2022-02-21T17:04:55.529 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package kubelet. 2022-02-21T17:04:55.542 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-02-21T17:04:55.651 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package kubelet. 2022-02-21T17:04:55.664 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-02-21T17:04:55.696 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking kubelet (1.23.4-00) ... 2022-02-21T17:04:55.701 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking kubelet (1.23.4-00) ... 2022-02-21T17:04:57.183 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package kubectl. 2022-02-21T17:04:57.196 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-02-21T17:04:57.232 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking kubectl (1.23.4-00) ... 2022-02-21T17:04:58.082 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package kubectl. 2022-02-21T17:04:58.094 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-02-21T17:04:58.112 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package kubectl. 2022-02-21T17:04:58.126 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-02-21T17:04:58.131 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking kubectl (1.23.4-00) ... 2022-02-21T17:04:58.170 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking kubectl (1.23.4-00) ... 2022-02-21T17:04:58.535 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package kubeadm. 2022-02-21T17:04:58.548 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-02-21T17:04:58.593 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-02-21T17:04:59.318 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package kubeadm. 2022-02-21T17:04:59.330 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-02-21T17:04:59.347 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package kubeadm. 2022-02-21T17:04:59.362 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-02-21T17:04:59.367 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-02-21T17:04:59.406 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-02-21T17:04:59.804 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-02-21T17:04:59.930 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up kubectl (1.23.4-00) ... 2022-02-21T17:05:00.056 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-02-21T17:05:00.514 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-02-21T17:05:00.519 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-02-21T17:05:00.643 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up kubectl (1.23.4-00) ... 2022-02-21T17:05:00.763 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up kubectl (1.23.4-00) ... 2022-02-21T17:05:00.768 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-02-21T17:05:00.889 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-02-21T17:05:00.925 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up socat (1.7.3.3-2) ... 2022-02-21T17:05:01.051 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cri-tools (1.19.0-00) ... 2022-02-21T17:05:01.278 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-02-21T17:05:01.387 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up kubelet (1.23.4-00) ... 2022-02-21T17:05:01.538 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up socat (1.7.3.3-2) ... 2022-02-21T17:05:01.616 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-02-21T17:05:01.664 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up cri-tools (1.19.0-00) ... 2022-02-21T17:05:01.733 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up socat (1.7.3.3-2) ... 2022-02-21T17:05:01.782 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-02-21T17:05:01.834 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up cri-tools (1.19.0-00) ... 2022-02-21T17:05:01.899 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up kubelet (1.23.4-00) ... 2022-02-21T17:05:01.960 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-02-21T17:05:02.086 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up kubelet (1.23.4-00) ... 2022-02-21T17:05:02.296 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-02-21T17:05:02.296 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up kubeadm (1.23.4-00) ... 2022-02-21T17:05:02.312 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-02-21T17:05:02.507 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T17:05:02.609 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up kubeadm (1.23.4-00) ... 2022-02-21T17:05:02.812 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up kubeadm (1.23.4-00) ... 2022-02-21T17:05:03.019 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T17:05:03.206 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-02-21T17:05:04.070 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-02-21T17:05:04.112 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-02-21T17:05:04.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-02-21T17:05:17.616 INFO:teuthology.orchestra.run.smithi067.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-02-21T17:05:17.745 INFO:teuthology.orchestra.run.smithi005.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-02-21T17:05:18.089 INFO:teuthology.orchestra.run.smithi033.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-02-21T17:05:20.869 INFO:teuthology.orchestra.run.smithi067.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-02-21T17:05:20.885 INFO:teuthology.orchestra.run.smithi005.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-02-21T17:05:21.135 INFO:teuthology.orchestra.run.smithi033.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-02-21T17:05:23.084 INFO:teuthology.orchestra.run.smithi005.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-02-21T17:05:23.416 INFO:teuthology.orchestra.run.smithi067.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-02-21T17:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-02-21T17:05:26.710 INFO:teuthology.orchestra.run.smithi005.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-02-21T17:05:27.424 INFO:teuthology.orchestra.run.smithi067.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-02-21T17:05:27.854 INFO:teuthology.orchestra.run.smithi033.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-02-21T17:05:28.100 INFO:teuthology.orchestra.run.smithi005.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-02-21T17:05:28.840 INFO:teuthology.orchestra.run.smithi067.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-02-21T17:05:29.220 INFO:teuthology.orchestra.run.smithi033.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-02-21T17:05:34.426 INFO:teuthology.orchestra.run.smithi005.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-02-21T17:05:35.527 INFO:teuthology.orchestra.run.smithi033.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-02-21T17:05:35.914 INFO:teuthology.orchestra.run.smithi067.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-02-21T17:05:36.416 INFO:teuthology.orchestra.run.smithi005.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-02-21T17:05:37.585 INFO:teuthology.orchestra.run.smithi033.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-02-21T17:05:38.264 INFO:teuthology.orchestra.run.smithi067.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-02-21T17:05:38.269 INFO:tasks.kubeadm:Token: abcdef.rbbs67gutu7s836j 2022-02-21T17:05:38.269 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): '172.21.15.5', Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): '172.21.15.33', Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): '172.21.15.67'} 2022-02-21T17:05:38.270 DEBUG:teuthology.orchestra.run.smithi005:> sudo kubeadm init --node-name smithi005 --token abcdef.rbbs67gutu7s836j --pod-network-cidr 10.248.32.0/21 2022-02-21T17:05:38.501 INFO:teuthology.orchestra.run.smithi005.stdout:[init] Using Kubernetes version: v1.23.4 2022-02-21T17:05:38.502 INFO:teuthology.orchestra.run.smithi005.stdout:[preflight] Running pre-flight checks 2022-02-21T17:05:38.781 INFO:teuthology.orchestra.run.smithi005.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-02-21T17:05:38.782 INFO:teuthology.orchestra.run.smithi005.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-02-21T17:05:38.782 INFO:teuthology.orchestra.run.smithi005.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-02-21T17:05:39.101 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-02-21T17:05:39.199 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "ca" certificate and key 2022-02-21T17:05:39.344 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "apiserver" certificate and key 2022-02-21T17:05:39.344 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi005] and IPs [10.96.0.1 172.21.15.5] 2022-02-21T17:05:39.474 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-02-21T17:05:39.782 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-02-21T17:05:40.025 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-02-21T17:05:40.201 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "etcd/ca" certificate and key 2022-02-21T17:05:40.253 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "etcd/server" certificate and key 2022-02-21T17:05:40.254 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi005] and IPs [172.21.15.5 127.0.0.1 ::1] 2022-02-21T17:05:40.465 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "etcd/peer" certificate and key 2022-02-21T17:05:40.466 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi005] and IPs [172.21.15.5 127.0.0.1 ::1] 2022-02-21T17:05:40.655 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-02-21T17:05:40.795 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-02-21T17:05:40.996 INFO:teuthology.orchestra.run.smithi005.stdout:[certs] Generating "sa" key and public key 2022-02-21T17:05:40.996 INFO:teuthology.orchestra.run.smithi005.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-02-21T17:05:41.279 INFO:teuthology.orchestra.run.smithi005.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-02-21T17:05:41.377 INFO:teuthology.orchestra.run.smithi005.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-02-21T17:05:41.643 INFO:teuthology.orchestra.run.smithi005.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-02-21T17:05:41.755 INFO:teuthology.orchestra.run.smithi005.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-02-21T17:05:41.770 INFO:teuthology.orchestra.run.smithi005.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-02-21T17:05:41.771 INFO:teuthology.orchestra.run.smithi005.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-02-21T17:05:41.771 INFO:teuthology.orchestra.run.smithi005.stdout:[kubelet-start] Starting the kubelet 2022-02-21T17:05:41.990 INFO:teuthology.orchestra.run.smithi005.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-02-21T17:05:41.990 INFO:teuthology.orchestra.run.smithi005.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-02-21T17:05:41.993 INFO:teuthology.orchestra.run.smithi005.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-02-21T17:05:41.994 INFO:teuthology.orchestra.run.smithi005.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-02-21T17:05:41.995 INFO:teuthology.orchestra.run.smithi005.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-02-21T17:05:41.997 INFO:teuthology.orchestra.run.smithi005.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-02-21T17:05:59.025 INFO:teuthology.orchestra.run.smithi005.stdout:[apiclient] All control plane components are healthy after 17.027541 seconds 2022-02-21T17:05:59.025 INFO:teuthology.orchestra.run.smithi005.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-02-21T17:05:59.053 INFO:teuthology.orchestra.run.smithi005.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-02-21T17:05:59.054 INFO:teuthology.orchestra.run.smithi005.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-02-21T17:05:59.612 INFO:teuthology.orchestra.run.smithi005.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-02-21T17:05:59.612 INFO:teuthology.orchestra.run.smithi005.stdout:[mark-control-plane] Marking the node smithi005 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-02-21T17:05:59.613 INFO:teuthology.orchestra.run.smithi005.stdout:[mark-control-plane] Marking the node smithi005 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-02-21T17:06:00.179 INFO:teuthology.orchestra.run.smithi005.stdout:[bootstrap-token] Using token: abcdef.rbbs67gutu7s836j 2022-02-21T17:06:00.179 INFO:teuthology.orchestra.run.smithi005.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-02-21T17:06:00.203 INFO:teuthology.orchestra.run.smithi005.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-02-21T17:06:00.232 INFO:teuthology.orchestra.run.smithi005.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-02-21T17:06:00.237 INFO:teuthology.orchestra.run.smithi005.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-02-21T17:06:00.245 INFO:teuthology.orchestra.run.smithi005.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-02-21T17:06:00.253 INFO:teuthology.orchestra.run.smithi005.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-02-21T17:06:00.328 INFO:teuthology.orchestra.run.smithi005.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-02-21T17:06:00.726 INFO:teuthology.orchestra.run.smithi005.stdout:[addons] Applied essential addon: CoreDNS 2022-02-21T17:06:00.821 INFO:teuthology.orchestra.run.smithi005.stdout:[addons] Applied essential addon: kube-proxy 2022-02-21T17:06:00.823 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:06:00.823 INFO:teuthology.orchestra.run.smithi005.stdout:Your Kubernetes control-plane has initialized successfully! 2022-02-21T17:06:00.824 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:06:00.824 INFO:teuthology.orchestra.run.smithi005.stdout:To start using your cluster, you need to run the following as a regular user: 2022-02-21T17:06:00.824 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:06:00.824 INFO:teuthology.orchestra.run.smithi005.stdout: mkdir -p $HOME/.kube 2022-02-21T17:06:00.825 INFO:teuthology.orchestra.run.smithi005.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-02-21T17:06:00.825 INFO:teuthology.orchestra.run.smithi005.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-02-21T17:06:00.825 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:06:00.825 INFO:teuthology.orchestra.run.smithi005.stdout:Alternatively, if you are the root user, you can run: 2022-02-21T17:06:00.826 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:06:00.826 INFO:teuthology.orchestra.run.smithi005.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-02-21T17:06:00.826 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:06:00.826 INFO:teuthology.orchestra.run.smithi005.stdout:You should now deploy a pod network to the cluster. 2022-02-21T17:06:00.826 INFO:teuthology.orchestra.run.smithi005.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-02-21T17:06:00.827 INFO:teuthology.orchestra.run.smithi005.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-02-21T17:06:00.827 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:06:00.827 INFO:teuthology.orchestra.run.smithi005.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-02-21T17:06:00.827 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:06:00.828 INFO:teuthology.orchestra.run.smithi005.stdout:kubeadm join 172.21.15.5:6443 --token abcdef.rbbs67gutu7s836j \ 2022-02-21T17:06:00.828 INFO:teuthology.orchestra.run.smithi005.stdout: --discovery-token-ca-cert-hash sha256:588bbb3c4e50255610ad37e991f8bdcad6ea1c80048b4923dbde211920425bea 2022-02-21T17:06:00.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo kubeadm join 172.21.15.5:6443 --node-name smithi033 --token abcdef.rbbs67gutu7s836j --discovery-token-unsafe-skip-ca-verification 2022-02-21T17:06:00.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo kubeadm join 172.21.15.5:6443 --node-name smithi067 --token abcdef.rbbs67gutu7s836j --discovery-token-unsafe-skip-ca-verification 2022-02-21T17:06:00.872 INFO:teuthology.orchestra.run.smithi033.stdout:[preflight] Running pre-flight checks 2022-02-21T17:06:00.875 INFO:teuthology.orchestra.run.smithi067.stdout:[preflight] Running pre-flight checks 2022-02-21T17:06:13.606 INFO:teuthology.orchestra.run.smithi033.stdout:[preflight] Reading configuration from the cluster... 2022-02-21T17:06:13.607 INFO:teuthology.orchestra.run.smithi033.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-02-21T17:06:13.752 INFO:teuthology.orchestra.run.smithi067.stdout:[preflight] Reading configuration from the cluster... 2022-02-21T17:06:13.752 INFO:teuthology.orchestra.run.smithi067.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-02-21T17:06:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:W0221 17:06:14.008109 10443 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-02-21T17:06:14.033 INFO:teuthology.orchestra.run.smithi033.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-02-21T17:06:14.033 INFO:teuthology.orchestra.run.smithi033.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-02-21T17:06:14.034 INFO:teuthology.orchestra.run.smithi033.stdout:[kubelet-start] Starting the kubelet 2022-02-21T17:06:14.240 INFO:teuthology.orchestra.run.smithi033.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-02-21T17:06:14.371 INFO:teuthology.orchestra.run.smithi067.stderr:W0221 17:06:14.366558 10553 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-02-21T17:06:14.444 INFO:teuthology.orchestra.run.smithi067.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-02-21T17:06:14.444 INFO:teuthology.orchestra.run.smithi067.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-02-21T17:06:14.444 INFO:teuthology.orchestra.run.smithi067.stdout:[kubelet-start] Starting the kubelet 2022-02-21T17:06:14.660 INFO:teuthology.orchestra.run.smithi067.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-02-21T17:06:19.780 INFO:teuthology.orchestra.run.smithi033.stdout: 2022-02-21T17:06:19.780 INFO:teuthology.orchestra.run.smithi033.stdout:This node has joined the cluster: 2022-02-21T17:06:19.780 INFO:teuthology.orchestra.run.smithi033.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-02-21T17:06:19.781 INFO:teuthology.orchestra.run.smithi033.stdout:* The Kubelet was informed of the new secure connection details. 2022-02-21T17:06:19.781 INFO:teuthology.orchestra.run.smithi033.stdout: 2022-02-21T17:06:19.781 INFO:teuthology.orchestra.run.smithi033.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-02-21T17:06:19.781 INFO:teuthology.orchestra.run.smithi033.stdout: 2022-02-21T17:06:20.186 INFO:teuthology.orchestra.run.smithi067.stdout: 2022-02-21T17:06:20.186 INFO:teuthology.orchestra.run.smithi067.stdout:This node has joined the cluster: 2022-02-21T17:06:20.187 INFO:teuthology.orchestra.run.smithi067.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-02-21T17:06:20.187 INFO:teuthology.orchestra.run.smithi067.stdout:* The Kubelet was informed of the new secure connection details. 2022-02-21T17:06:20.187 INFO:teuthology.orchestra.run.smithi067.stdout: 2022-02-21T17:06:20.187 INFO:teuthology.orchestra.run.smithi067.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-02-21T17:06:20.188 INFO:teuthology.orchestra.run.smithi067.stdout: 2022-02-21T17:06:20.193 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:06:20.194 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-02-21T17:06:20.209 INFO:tasks.kubeadm:Setting up kubectl 2022-02-21T17:06:20.210 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-02-21T17:06:20.273 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-02-21T17:06:20.292 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-02-21T17:06:20.311 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:06:20.311 DEBUG:teuthology.orchestra.run.smithi005:> dd of=.kube/config 2022-02-21T17:06:20.319 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:06:20.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/root/.kube/config 2022-02-21T17:06:20.378 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:06:20.379 DEBUG:teuthology.orchestra.run.smithi033:> dd of=.kube/config 2022-02-21T17:06:20.386 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:06:20.387 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/root/.kube/config 2022-02-21T17:06:20.400 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:06:20.400 DEBUG:teuthology.orchestra.run.smithi067:> dd of=.kube/config 2022-02-21T17:06:20.408 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:06:20.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/root/.kube/config 2022-02-21T17:06:20.424 DEBUG:teuthology.orchestra.run.smithi005:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-02-21T17:06:20.440 INFO:teuthology.orchestra.run.smithi005.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-02-21T17:06:20.440 INFO:teuthology.orchestra.run.smithi005.stderr: Dload Upload Total Spent Left Speed 2022-02-21T17:06:20.554 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2022-02-21T17:06:20.554 INFO:teuthology.orchestra.run.smithi005.stdout:apiVersion: policy/v1beta1 2022-02-21T17:06:20.554 INFO:teuthology.orchestra.run.smithi005.stdout:kind: PodSecurityPolicy 2022-02-21T17:06:20.555 INFO:teuthology.orchestra.run.smithi005.stdout:metadata: 2022-02-21T17:06:20.555 INFO:teuthology.orchestra.run.smithi005.stdout: name: psp.flannel.unprivileged 2022-02-21T17:06:20.555 INFO:teuthology.orchestra.run.smithi005.stdout: annotations: 2022-02-21T17:06:20.555 INFO:teuthology.orchestra.run.smithi005.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-02-21T17:06:20.556 INFO:teuthology.orchestra.run.smithi005.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-02-21T17:06:20.556 INFO:teuthology.orchestra.run.smithi005.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-02-21T17:06:20.556 INFO:teuthology.orchestra.run.smithi005.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-02-21T17:06:20.556 INFO:teuthology.orchestra.run.smithi005.stdout:spec: 2022-02-21T17:06:20.557 INFO:teuthology.orchestra.run.smithi005.stdout: privileged: false 2022-02-21T17:06:20.557 INFO:teuthology.orchestra.run.smithi005.stdout: volumes: 2022-02-21T17:06:20.557 INFO:teuthology.orchestra.run.smithi005.stdout: - configMap 2022-02-21T17:06:20.557 INFO:teuthology.orchestra.run.smithi005.stdout: - secret 2022-02-21T17:06:20.558 INFO:teuthology.orchestra.run.smithi005.stdout: - emptyDir 2022-02-21T17:06:20.558 INFO:teuthology.orchestra.run.smithi005.stdout: - hostPath 2022-02-21T17:06:20.558 INFO:teuthology.orchestra.run.smithi005.stdout: allowedHostPaths: 2022-02-21T17:06:20.558 INFO:teuthology.orchestra.run.smithi005.stdout: - pathPrefix: "/etc/cni/net.d" 2022-02-21T17:06:20.559 INFO:teuthology.orchestra.run.smithi005.stdout: - pathPrefix: "/etc/kube-flannel" 2022-02-21T17:06:20.559 INFO:teuthology.orchestra.run.smithi005.stdout: - pathPrefix: "/run/flannel" 2022-02-21T17:06:20.559 INFO:teuthology.orchestra.run.smithi005.stdout: readOnlyRootFilesystem: false 2022-02-21T17:06:20.559 INFO:teuthology.orchestra.run.smithi005.stdout: # Users and groups 2022-02-21T17:06:20.559 INFO:teuthology.orchestra.run.smithi005.stdout: runAsUser: 2022-02-21T17:06:20.560 INFO:teuthology.orchestra.run.smithi005.stdout: rule: RunAsAny 2022-02-21T17:06:20.560 INFO:teuthology.orchestra.run.smithi005.stdout: supplementalGroups: 2022-02-21T17:06:20.560 INFO:teuthology.orchestra.run.smithi005.stdout: rule: RunAsAny 2022-02-21T17:06:20.560 INFO:teuthology.orchestra.run.smithi005.stdout: fsGroup: 2022-02-21T17:06:20.561 INFO:teuthology.orchestra.run.smithi005.stdout: rule: RunAsAny 2022-02-21T17:06:20.561 INFO:teuthology.orchestra.run.smithi005.stdout: # Privilege Escalation 2022-02-21T17:06:20.561 INFO:teuthology.orchestra.run.smithi005.stdout: allowPrivilegeEscalation: false 2022-02-21T17:06:20.561 INFO:teuthology.orchestra.run.smithi005.stdout: defaultAllowPrivilegeEscalation: false 2022-02-21T17:06:20.561 INFO:teuthology.orchestra.run.smithi005.stdout: # Capabilities 2022-02-21T17:06:20.562 INFO:teuthology.orchestra.run.smithi005.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-02-21T17:06:20.562 INFO:teuthology.orchestra.run.smithi005.stdout: defaultAddCapabilities: [] 2022-02-21T17:06:20.562 INFO:teuthology.orchestra.run.smithi005.stdout: requiredDropCapabilities: [] 2022-02-21T17:06:20.562 INFO:teuthology.orchestra.run.smithi005.stdout: # Host namespaces 2022-02-21T17:06:20.563 INFO:teuthology.orchestra.run.smithi005.stdout: hostPID: false 2022-02-21T17:06:20.563 INFO:teuthology.orchestra.run.smithi005.stdout: hostIPC: false 2022-02-21T17:06:20.563 INFO:teuthology.orchestra.run.smithi005.stdout: hostNetwork: true 2022-02-21T17:06:20.563 INFO:teuthology.orchestra.run.smithi005.stdout: hostPorts: 2022-02-21T17:06:20.563 INFO:teuthology.orchestra.run.smithi005.stdout: - min: 0 2022-02-21T17:06:20.564 INFO:teuthology.orchestra.run.smithi005.stdout: max: 65535 2022-02-21T17:06:20.564 INFO:teuthology.orchestra.run.smithi005.stdout: # SELinux 2022-02-21T17:06:20.564 INFO:teuthology.orchestra.run.smithi005.stdout: seLinux: 2022-02-21T17:06:20.564 INFO:teuthology.orchestra.run.smithi005.stdout: # SELinux is unused in CaaSP 2022-02-21T17:06:20.565 INFO:teuthology.orchestra.run.smithi005.stdout: rule: 'RunAsAny' 2022-02-21T17:06:20.565 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2022-02-21T17:06:20.565 INFO:teuthology.orchestra.run.smithi005.stdout:kind: ClusterRole 2022-02-21T17:06:20.565 INFO:teuthology.orchestra.run.smithi005.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-02-21T17:06:20.566 INFO:teuthology.orchestra.run.smithi005.stdout:metadata: 2022-02-21T17:06:20.566 INFO:teuthology.orchestra.run.smithi005.stdout: name: flannel 2022-02-21T17:06:20.566 INFO:teuthology.orchestra.run.smithi005.stdout:rules: 2022-02-21T17:06:20.566 INFO:teuthology.orchestra.run.smithi005.stdout:- apiGroups: ['extensions'] 2022-02-21T17:06:20.567 INFO:teuthology.orchestra.run.smithi005.stdout: resources: ['podsecuritypolicies'] 2022-02-21T17:06:20.567 INFO:teuthology.orchestra.run.smithi005.stdout: verbs: ['use'] 2022-02-21T17:06:20.567 INFO:teuthology.orchestra.run.smithi005.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-02-21T17:06:20.567 INFO:teuthology.orchestra.run.smithi005.stdout:- apiGroups: 2022-02-21T17:06:20.567 INFO:teuthology.orchestra.run.smithi005.stdout: - "" 2022-02-21T17:06:20.568 INFO:teuthology.orchestra.run.smithi005.stdout: resources: 2022-02-21T17:06:20.568 INFO:teuthology.orchestra.run.smithi005.stdout: - pods 2022-02-21T17:06:20.568 INFO:teuthology.orchestra.run.smithi005.stdout: verbs: 2022-02-21T17:06:20.568 INFO:teuthology.orchestra.run.smithi005.stdout: - get 2022-02-21T17:06:20.569 INFO:teuthology.orchestra.run.smithi005.stdout:- apiGroups: 2022-02-21T17:06:20.569 INFO:teuthology.orchestra.run.smithi005.stdout: - "" 2022-02-21T17:06:20.569 INFO:teuthology.orchestra.run.smithi005.stdout: resources: 2022-02-21T17:06:20.569 INFO:teuthology.orchestra.run.smithi005.stdout: - nodes 2022-02-21T17:06:20.570 INFO:teuthology.orchestra.run.smithi005.stdout: verbs: 2022-02-21T17:06:20.570 INFO:teuthology.orchestra.run.smithi005.stdout: - list 2022-02-21T17:06:20.570 INFO:teuthology.orchestra.run.smithi005.stdout: - watch 2022-02-21T17:06:20.570 INFO:teuthology.orchestra.run.smithi005.stdout:- apiGroups: 2022-02-21T17:06:20.571 INFO:teuthology.orchestra.run.smithi005.stdout: - "" 2022-02-21T17:06:20.571 INFO:teuthology.orchestra.run.smithi005.stdout: resources: 2022-02-21T17:06:20.571 INFO:teuthology.orchestra.run.smithi005.stdout: - nodes/status 2022-02-21T17:06:20.571 INFO:teuthology.orchestra.run.smithi005.stdout: verbs: 2022-02-21T17:06:20.572 INFO:teuthology.orchestra.run.smithi005.stdout: - patch 2022-02-21T17:06:20.572 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2022-02-21T17:06:20.572 INFO:teuthology.orchestra.run.smithi005.stdout:kind: ClusterRoleBinding 2022-02-21T17:06:20.572 INFO:teuthology.orchestra.run.smithi005.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-02-21T17:06:20.572 INFO:teuthology.orchestra.run.smithi005.stdout:metadata: 2022-02-21T17:06:20.573 INFO:teuthology.orchestra.run.smithi005.stdout: name: flannel 2022-02-21T17:06:20.573 INFO:teuthology.orchestra.run.smithi005.stdout:roleRef: 2022-02-21T17:06:20.573 INFO:teuthology.orchestra.run.smithi005.stdout: apiGroup: rbac.authorization.k8s.io 2022-02-21T17:06:20.573 INFO:teuthology.orchestra.run.smithi005.stdout: kind: ClusterRole 2022-02-21T17:06:20.574 INFO:teuthology.orchestra.run.smithi005.stdout: name: flannel 2022-02-21T17:06:20.574 INFO:teuthology.orchestra.run.smithi005.stdout:subjects: 2022-02-21T17:06:20.574 INFO:teuthology.orchestra.run.smithi005.stdout:- kind: ServiceAccount 2022-02-21T17:06:20.574 INFO:teuthology.orchestra.run.smithi005.stdout: name: flannel 2022-02-21T17:06:20.574 INFO:teuthology.orchestra.run.smithi005.stdout: namespace: kube-system 2022-02-21T17:06:20.575 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2022-02-21T17:06:20.575 INFO:teuthology.orchestra.run.smithi005.stdout:apiVersion: v1 2022-02-21T17:06:20.575 INFO:teuthology.orchestra.run.smithi005.stdout:kind: ServiceAccount 2022-02-21T17:06:20.575 INFO:teuthology.orchestra.run.smithi005.stdout:metadata: 2022-02-21T17:06:20.576 INFO:teuthology.orchestra.run.smithi005.stdout: name: flannel 2022-02-21T17:06:20.576 INFO:teuthology.orchestra.run.smithi005.stdout: namespace: kube-system 2022-02-21T17:06:20.576 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2022-02-21T17:06:20.576 INFO:teuthology.orchestra.run.smithi005.stdout:kind: ConfigMap 2022-02-21T17:06:20.576 INFO:teuthology.orchestra.run.smithi005.stdout:apiVersion: v1 2022-02-21T17:06:20.577 INFO:teuthology.orchestra.run.smithi005.stdout:metadata: 2022-02-21T17:06:20.577 INFO:teuthology.orchestra.run.smithi005.stdout: name: kube-flannel-cfg 2022-02-21T17:06:20.577 INFO:teuthology.orchestra.run.smithi005.stdout: namespace: kube-system 2022-02-21T17:06:20.577 INFO:teuthology.orchestra.run.smithi005.stdout: labels: 2022-02-21T17:06:20.578 INFO:teuthology.orchestra.run.smithi005.stdout: tier: node 2022-02-21T17:06:20.578 INFO:teuthology.orchestra.run.smithi005.stdout: app: flannel 2022-02-21T17:06:20.578 INFO:teuthology.orchestra.run.smithi005.stdout:data: 2022-02-21T17:06:20.578 INFO:teuthology.orchestra.run.smithi005.stdout: cni-conf.json: | 2022-02-21T17:06:20.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-02-21T17:06:20.579 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "cbr0", 2022-02-21T17:06:20.579 INFO:teuthology.orchestra.run.smithi005.stdout: "cniVersion": "0.3.1", 2022-02-21T17:06:20.579 INFO:teuthology.orchestra.run.smithi005.stdout: "plugins": [ 2022-02-21T17:06:20.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-02-21T17:06:20.580 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "flannel", 2022-02-21T17:06:20.580 INFO:teuthology.orchestra.run.smithi005.stdout: "delegate": { 2022-02-21T17:06:20.580 INFO:teuthology.orchestra.run.smithi005.stdout: "hairpinMode": true, 2022-02-21T17:06:20.580 INFO:teuthology.orchestra.run.smithi005.stdout: "isDefaultGateway": true 2022-02-21T17:06:20.581 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-02-21T17:06:20.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2022-02-21T17:06:20.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-02-21T17:06:20.581 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "portmap", 2022-02-21T17:06:20.581 INFO:teuthology.orchestra.run.smithi005.stdout: "capabilities": { 2022-02-21T17:06:20.582 INFO:teuthology.orchestra.run.smithi005.stdout: "portMappings": true 2022-02-21T17:06:20.582 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-02-21T17:06:20.582 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-02-21T17:06:20.582 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2022-02-21T17:06:20.583 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-02-21T17:06:20.583 INFO:teuthology.orchestra.run.smithi005.stdout: net-conf.json: | 2022-02-21T17:06:20.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2022-02-21T17:06:20.583 INFO:teuthology.orchestra.run.smithi005.stdout: "Network": "10.244.0.0/16", 2022-02-21T17:06:20.583 INFO:teuthology.orchestra.run.smithi005.stdout: "Backend": { 2022-02-21T17:06:20.584 INFO:teuthology.orchestra.run.smithi005.stdout: "Type": "vxlan" 2022-02-21T17:06:20.584 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-02-21T17:06:20.584 INFO:teuthology.orchestra.run.smithi005.stdout: } 2022-02-21T17:06:20.584 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2022-02-21T17:06:20.585 INFO:teuthology.orchestra.run.smithi005.stdout:apiVersion: apps/v1 2022-02-21T17:06:20.585 INFO:teuthology.orchestra.run.smithi005.stdout:kind: DaemonSet 2022-02-21T17:06:20.585 INFO:teuthology.orchestra.run.smithi005.stdout:metadata: 2022-02-21T17:06:20.585 INFO:teuthology.orchestra.run.smithi005.stdout: name: kube-flannel-ds 2022-02-21T17:06:20.586 INFO:teuthology.orchestra.run.smithi005.stdout: namespace: kube-system 2022-02-21T17:06:20.586 INFO:teuthology.orchestra.run.smithi005.stdout: labels: 2022-02-21T17:06:20.586 INFO:teuthology.orchestra.run.smithi005.stdout: tier: node 2022-02-21T17:06:20.586 INFO:teuthology.orchestra.run.smithi005.stdout: app: flannel 2022-02-21T17:06:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:spec: 2022-02-21T17:06:20.587 INFO:teuthology.orchestra.run.smithi005.stdout: selector: 2022-02-21T17:06:20.587 INFO:teuthology.orchestra.run.smithi005.stdout: matchLabels: 2022-02-21T17:06:20.587 INFO:teuthology.orchestra.run.smithi005.stdout: app: flannel 2022-02-21T17:06:20.587 INFO:teuthology.orchestra.run.smithi005.stdout: template: 2022-02-21T17:06:20.587 INFO:teuthology.orchestra.run.smithi005.stdout: metadata: 2022-02-21T17:06:20.588 INFO:teuthology.orchestra.run.smithi005.stdout: labels: 2022-02-21T17:06:20.588 INFO:teuthology.orchestra.run.smithi005.stdout: tier: node 2022-02-21T17:06:20.588 INFO:teuthology.orchestra.run.smithi005.stdout: app: flannel 2022-02-21T17:06:20.588 INFO:teuthology.orchestra.run.smithi005.stdout: spec: 2022-02-21T17:06:20.589 INFO:teuthology.orchestra.run.smithi005.stdout: affinity: 2022-02-21T17:06:20.589 INFO:teuthology.orchestra.run.smithi005.stdout: nodeAffinity: 2022-02-21T17:06:20.589 INFO:teuthology.orchestra.run.smithi005.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-02-21T17:06:20.589 INFO:teuthology.orchestra.run.smithi005.stdout: nodeSelectorTerms: 2022-02-21T17:06:20.589 INFO:teuthology.orchestra.run.smithi005.stdout: - matchExpressions: 2022-02-21T17:06:20.590 INFO:teuthology.orchestra.run.smithi005.stdout: - key: kubernetes.io/os 2022-02-21T17:06:20.590 INFO:teuthology.orchestra.run.smithi005.stdout: operator: In 2022-02-21T17:06:20.590 INFO:teuthology.orchestra.run.smithi005.stdout: values: 2022-02-21T17:06:20.590 INFO:teuthology.orchestra.run.smithi005.stdout: - linux 2022-02-21T17:06:20.591 INFO:teuthology.orchestra.run.smithi005.stdout: hostNetwork: true 2022-02-21T17:06:20.591 INFO:teuthology.orchestra.run.smithi005.stdout: priorityClassName: system-node-critical 2022-02-21T17:06:20.591 INFO:teuthology.orchestra.run.smithi005.stdout: tolerations: 2022-02-21T17:06:20.591 INFO:teuthology.orchestra.run.smithi005.stdout: - operator: Exists 2022-02-21T17:06:20.591 INFO:teuthology.orchestra.run.smithi005.stdout: effect: NoSchedule 2022-02-21T17:06:20.592 INFO:teuthology.orchestra.run.smithi005.stdout: serviceAccountName: flannel 2022-02-21T17:06:20.592 INFO:teuthology.orchestra.run.smithi005.stdout: initContainers: 2022-02-21T17:06:20.592 INFO:teuthology.orchestra.run.smithi005.stdout: - name: install-cni-plugin 2022-02-21T17:06:20.593 INFO:teuthology.orchestra.run.smithi005.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-02-21T17:06:20.593 INFO:teuthology.orchestra.run.smithi005.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-02-21T17:06:20.594 INFO:teuthology.orchestra.run.smithi005.stdout: command: 2022-02-21T17:06:20.594 INFO:teuthology.orchestra.run.smithi005.stdout: - cp 2022-02-21T17:06:20.594 INFO:teuthology.orchestra.run.smithi005.stdout: args: 2022-02-21T17:06:20.595 INFO:teuthology.orchestra.run.smithi005.stdout: - -f 2022-02-21T17:06:20.595 INFO:teuthology.orchestra.run.smithi005.stdout: - /flannel 2022-02-21T17:06:20.595 INFO:teuthology.orchestra.run.smithi005.stdout: - /opt/cni/bin/flannel 2022-02-21T17:06:20.595 INFO:teuthology.orchestra.run.smithi005.stdout: volumeMounts: 2022-02-21T17:06:20.596 INFO:teuthology.orchestra.run.smithi005.stdout: - name: cni-plugin 2022-02-21T17:06:20.596 INFO:teuthology.orchestra.run.smithi005.stdout: mountPath: /opt/cni/bin 2022-02-21T17:06:20.596 INFO:teuthology.orchestra.run.smithi005.stdout: - name: install-cni 2022-02-21T17:06:20.596 INFO:teuthology.orchestra.run.smithi005.stdout: #image: flannelcni/flannel:v0.16.3 for ppc64le and mips64le (dockerhub limitations may apply) 2022-02-21T17:06:20.597 INFO:teuthology.orchestra.run.smithi005.stdout: image: rancher/mirrored-flannelcni-flannel:v0.16.3 2022-02-21T17:06:20.597 INFO:teuthology.orchestra.run.smithi005.stdout: command: 2022-02-21T17:06:20.597 INFO:teuthology.orchestra.run.smithi005.stdout: - cp 2022-02-21T17:06:20.597 INFO:teuthology.orchestra.run.smithi005.stdout: args: 2022-02-21T17:06:20.598 INFO:teuthology.orchestra.run.smithi005.stdout: - -f 2022-02-21T17:06:20.598 INFO:teuthology.orchestra.run.smithi005.stdout: - /etc/kube-flannel/cni-conf.json 2022-02-21T17:06:20.599 INFO:teuthology.orchestra.run.smithi005.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-02-21T17:06:20.599 INFO:teuthology.orchestra.run.smithi005.stdout: volumeMounts: 2022-02-21T17:06:20.600 INFO:teuthology.orchestra.run.smithi005.stdout: - name: cni 2022-02-21T17:06:20.600 INFO:teuthology.orchestra.run.smithi005.stdout: mountPath: /etc/cni/net.d 2022-02-21T17:06:20.600 INFO:teuthology.orchestra.run.smithi005.stdout: - name: flannel-cfg 2022-02-21T17:06:20.600 INFO:teuthology.orchestra.run.smithi005.stdout: mountPath: /etc/kube-flannel/ 2022-02-21T17:06:20.601 INFO:teuthology.orchestra.run.smithi005.stdout: containers: 2022-02-21T17:06:20.601 INFO:teuthology.orchestra.run.smithi005.stdout: - name: kube-flannel 2022-02-21T17:06:20.601 INFO:teuthology.orchestra.run.smithi005.stdout: #image: flannelcni/flannel:v0.16.3 for ppc64le and mips64le (dockerhub limitations may apply) 2022-02-21T17:06:20.601 INFO:teuthology.orchestra.run.smithi005.stdout: image: rancher/mirrored-flannelcni-flannel:v0.16.3 2022-02-21T17:06:20.602 INFO:teuthology.orchestra.run.smithi005.stdout: command: 2022-02-21T17:06:20.602 INFO:teuthology.orchestra.run.smithi005.stdout: - /opt/bin/flanneld 2022-02-21T17:06:20.602 INFO:teuthology.orchestra.run.smithi005.stdout: args: 2022-02-21T17:06:20.602 INFO:teuthology.orchestra.run.smithi005.stdout: - --ip-masq 2022-02-21T17:06:20.603 INFO:teuthology.orchestra.run.smithi005.stdout: - --kube-subnet-mgr 2022-02-21T17:06:20.603 INFO:teuthology.orchestra.run.smithi005.stdout: resources: 2022-02-21T17:06:20.603 INFO:teuthology.orchestra.run.smithi005.stdout: requests: 2022-02-21T17:06:20.604 INFO:teuthology.orchestra.run.smithi005.stdout: cpu: "100m" 2022-02-21T17:06:20.604 INFO:teuthology.orchestra.run.smithi005.stdout: memory: "50Mi" 2022-02-21T17:06:20.604 INFO:teuthology.orchestra.run.smithi005.stdout: limits: 2022-02-21T17:06:20.604 INFO:teuthology.orchestra.run.smithi005.stdout: cpu: "100m" 2022-02-21T17:06:20.604 INFO:teuthology.orchestra.run.smithi005.stdout: memory: "50Mi" 2022-02-21T17:06:20.605 INFO:teuthology.orchestra.run.smithi005.stdout: securityContext: 2022-02-21T17:06:20.605 INFO:teuthology.orchestra.run.smithi005.stdout: privileged: false 2022-02-21T17:06:20.605 INFO:teuthology.orchestra.run.smithi005.stdout: capabilities: 2022-02-21T17:06:20.605 INFO:teuthology.orchestra.run.smithi005.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-02-21T17:06:20.605 INFO:teuthology.orchestra.run.smithi005.stdout: env: 2022-02-21T17:06:20.606 INFO:teuthology.orchestra.run.smithi005.stdout: - name: POD_NAME 2022-02-21T17:06:20.606 INFO:teuthology.orchestra.run.smithi005.stdout: valueFrom: 2022-02-21T17:06:20.606 INFO:teuthology.orchestra.run.smithi005.stdout: fieldRef: 2022-02-21T17:06:20.606 INFO:teuthology.orchestra.run.smithi005.stdout: fieldPath: metadata.name 2022-02-21T17:06:20.607 INFO:teuthology.orchestra.run.smithi005.stdout: - name: POD_NAMESPACE 2022-02-21T17:06:20.607 INFO:teuthology.orchestra.run.smithi005.stdout: valueFrom: 2022-02-21T17:06:20.607 INFO:teuthology.orchestra.run.smithi005.stdout: fieldRef: 2022-02-21T17:06:20.607 INFO:teuthology.orchestra.run.smithi005.stdout: fieldPath: metadata.namespace 2022-02-21T17:06:20.608 INFO:teuthology.orchestra.run.smithi005.stdout: volumeMounts: 2022-02-21T17:06:20.608 INFO:teuthology.orchestra.run.smithi005.stdout: - name: run 2022-02-21T17:06:20.608 INFO:teuthology.orchestra.run.smithi005.stdout: mountPath: /run/flannel 2022-02-21T17:06:20.608 INFO:teuthology.orchestra.run.smithi005.stdout: - name: flannel-cfg 2022-02-21T17:06:20.608 INFO:teuthology.orchestra.run.smithi005.stdout: mountPath: /etc/kube-flannel/ 2022-02-21T17:06:20.609 INFO:teuthology.orchestra.run.smithi005.stdout: - name: xtables-lock 2022-02-21T17:06:20.609 INFO:teuthology.orchestra.run.smithi005.stdout: mountPath: /run/xtables.lock 2022-02-21T17:06:20.609 INFO:teuthology.orchestra.run.smithi005.stdout: volumes: 2022-02-21T17:06:20.609 INFO:teuthology.orchestra.run.smithi005.stdout: - name: run 2022-02-21T17:06:20.610 INFO:teuthology.orchestra.run.smithi005.stdout: hostPath: 2022-02-21T17:06:20.610 INFO:teuthology.orchestra.run.smithi005.stdout: path: /run/flannel 2022-02-21T17:06:20.610 INFO:teuthology.orchestra.run.smithi005.stdout: - name: cni-plugin 2022-02-21T17:06:20.610 INFO:teuthology.orchestra.run.smithi005.stdout: hostPath: 2022-02-21T17:06:20.610 INFO:teuthology.orchestra.run.smithi005.stdout: path: /opt/cni/bin 2022-02-21T17:06:20.611 INFO:teuthology.orchestra.run.smithi005.stdout: - name: cni 2022-02-21T17:06:20.611 INFO:teuthology.orchestra.run.smithi005.stdout: hostPath: 2022-02-21T17:06:20.611 INFO:teuthology.orchestra.run.smithi005.stdout: path: /etc/cni/net.d 2022-02-21T17:06:20.611 INFO:teuthology.orchestra.run.smithi005.stdout: - name: flannel-cfg 2022-02-21T17:06:20.612 INFO:teuthology.orchestra.run.smithi005.stdout: configMap: 2022-02-21T17:06:20.612 INFO:teuthology.orchestra.run.smithi005.stdout: name: kube-flannel-cfg 2022-02-21T17:06:20.612 INFO:teuthology.orchestra.run.smithi005.stdout: - name: xtables-lock 2022-02-21T17:06:20.613 INFO:teuthology.orchestra.run.smithi005.stdout: hostPath: 2022-02-21T17:06:20.613 INFO:teuthology.orchestra.run.smithi005.stdout: path: /run/xtables.lock 2022-02-21T17:06:20.613 INFO:teuthology.orchestra.run.smithi005.stdout: type: FileOrCreate 2022-02-21T17:06:20.614 INFO:teuthology.orchestra.run.smithi005.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 49929 0 --:--:-- --:--:-- --:--:-- 49929 2022-02-21T17:06:20.642 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-02-21T17:06:20.642 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.248.32.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-02-21T17:06:20.656 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.248.32.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.16.3 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.16.3 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-02-21T17:06:20.657 DEBUG:teuthology.orchestra.run.smithi005:> kubectl apply -f - 2022-02-21T17:06:22.937 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-02-21T17:06:22.944 INFO:teuthology.orchestra.run.smithi005.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-02-21T17:06:22.983 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-02-21T17:06:22.989 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-02-21T17:06:22.998 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/flannel created 2022-02-21T17:06:23.023 INFO:teuthology.orchestra.run.smithi005.stdout:configmap/kube-flannel-cfg created 2022-02-21T17:06:23.062 INFO:teuthology.orchestra.run.smithi005.stdout:daemonset.apps/kube-flannel-ds created 2022-02-21T17:06:23.067 INFO:tasks.kubeadm:Scanning for scratch devices 2022-02-21T17:06:23.067 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:06:23.068 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:06:23.073 INFO:tasks.kubeadm:Scratch devices on smithi005: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T17:06:23.073 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-02-21T17:06:23.101 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records in 2022-02-21T17:06:23.101 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2022-02-21T17:06:23.101 INFO:teuthology.orchestra.run.smithi005.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0132226 s, 793 MB/s 2022-02-21T17:06:23.103 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-02-21T17:06:23.124 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records in 2022-02-21T17:06:23.125 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2022-02-21T17:06:23.125 INFO:teuthology.orchestra.run.smithi005.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121428 s, 864 MB/s 2022-02-21T17:06:23.126 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-02-21T17:06:23.151 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records in 2022-02-21T17:06:23.151 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2022-02-21T17:06:23.151 INFO:teuthology.orchestra.run.smithi005.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0142904 s, 734 MB/s 2022-02-21T17:06:23.152 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-02-21T17:06:23.219 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records in 2022-02-21T17:06:23.219 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2022-02-21T17:06:23.219 INFO:teuthology.orchestra.run.smithi005.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0101836 s, 1.0 GB/s 2022-02-21T17:06:23.221 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:06:23.221 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:06:23.229 INFO:tasks.kubeadm:Scratch devices on smithi033: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T17:06:23.229 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-02-21T17:06:23.303 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records in 2022-02-21T17:06:23.303 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2022-02-21T17:06:23.304 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0177684 s, 590 MB/s 2022-02-21T17:06:23.305 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-02-21T17:06:23.335 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records in 2022-02-21T17:06:23.335 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2022-02-21T17:06:23.336 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163021 s, 643 MB/s 2022-02-21T17:06:23.337 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-02-21T17:06:23.365 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records in 2022-02-21T17:06:23.366 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2022-02-21T17:06:23.366 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0164858 s, 636 MB/s 2022-02-21T17:06:23.367 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-02-21T17:06:23.394 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records in 2022-02-21T17:06:23.394 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2022-02-21T17:06:23.394 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.016192 s, 648 MB/s 2022-02-21T17:06:23.396 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:06:23.396 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:06:23.404 INFO:tasks.kubeadm:Scratch devices on smithi067: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T17:06:23.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-02-21T17:06:23.430 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records in 2022-02-21T17:06:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2022-02-21T17:06:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0154328 s, 679 MB/s 2022-02-21T17:06:23.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-02-21T17:06:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records in 2022-02-21T17:06:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2022-02-21T17:06:23.460 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0154237 s, 680 MB/s 2022-02-21T17:06:23.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-02-21T17:06:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records in 2022-02-21T17:06:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2022-02-21T17:06:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0183491 s, 571 MB/s 2022-02-21T17:06:23.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-02-21T17:06:23.567 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records in 2022-02-21T17:06:23.567 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2022-02-21T17:06:23.567 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0159154 s, 659 MB/s 2022-02-21T17:06:23.588 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-02-21T17:06:23.589 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi005-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi005 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi005-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi005 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi005-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi005 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi005-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi005 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi033-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi033 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi033-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi033 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi033-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi033 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi033-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi033 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi067-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi067 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi067-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi067 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi067-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi067 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi067-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi067 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-02-21T17:06:23.589 DEBUG:teuthology.orchestra.run.smithi005:> kubectl create -f - 2022-02-21T17:06:24.169 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi005-lv-1 created 2022-02-21T17:06:24.178 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi005-lv-2 created 2022-02-21T17:06:24.298 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi005-lv-3 created 2022-02-21T17:06:24.336 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi005-lv-4 created 2022-02-21T17:06:24.365 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi033-lv-1 created 2022-02-21T17:06:24.379 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi033-lv-2 created 2022-02-21T17:06:24.457 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi033-lv-3 created 2022-02-21T17:06:24.462 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi033-lv-4 created 2022-02-21T17:06:24.516 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi067-lv-1 created 2022-02-21T17:06:24.534 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi067-lv-2 created 2022-02-21T17:06:24.598 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi067-lv-3 created 2022-02-21T17:06:24.602 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi067-lv-4 created 2022-02-21T17:06:24.629 INFO:teuthology.orchestra.run.smithi005.stdout:storageclass.storage.k8s.io/scratch created 2022-02-21T17:06:24.634 DEBUG:teuthology.orchestra.run.smithi005:> kubectl taint node smithi005 node-role.kubernetes.io/master- || true 2022-02-21T17:06:24.806 INFO:teuthology.orchestra.run.smithi005.stdout:node/smithi005 untainted 2022-02-21T17:06:24.808 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-02-21T17:06:24.808 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-02-21T17:06:24.820 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-02-21T17:06:24.821 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:06:24.822 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:06:24.827 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T17:06:24.828 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-02-21T17:06:24.877 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-02-21T17:06:24.878 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:24.878 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-02-21T17:06:24.878 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:24.879 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-02-21 17:06:23.098508823 +0000 2022-02-21T17:06:24.879 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-02-21 17:06:23.094508823 +0000 2022-02-21T17:06:24.879 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-02-21 17:06:23.094508823 +0000 2022-02-21T17:06:24.879 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-02-21T17:06:24.884 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-02-21T17:06:24.884 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-02-21T17:06:24.884 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000197262 s, 2.6 MB/s 2022-02-21T17:06:24.888 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-02-21T17:06:24.894 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-02-21T17:06:24.895 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:24.895 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-02-21T17:06:24.895 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:24.895 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-02-21 17:06:23.122508823 +0000 2022-02-21T17:06:24.896 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-02-21 17:06:23.122508823 +0000 2022-02-21T17:06:24.896 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-02-21 17:06:23.122508823 +0000 2022-02-21T17:06:24.896 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-02-21T17:06:24.901 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-02-21T17:06:24.901 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-02-21T17:06:24.901 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000209838 s, 2.4 MB/s 2022-02-21T17:06:24.905 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-02-21T17:06:24.911 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-02-21T17:06:24.912 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:24.912 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-02-21T17:06:24.912 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:24.912 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-02-21 17:06:23.146508822 +0000 2022-02-21T17:06:24.913 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-02-21 17:06:23.142508822 +0000 2022-02-21T17:06:24.913 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-02-21 17:06:23.142508822 +0000 2022-02-21T17:06:24.913 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-02-21T17:06:24.919 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-02-21T17:06:24.920 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-02-21T17:06:24.920 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000189146 s, 2.7 MB/s 2022-02-21T17:06:24.923 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-02-21T17:06:24.928 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-02-21T17:06:24.929 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:24.929 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-02-21T17:06:24.929 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:24.929 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-02-21 17:06:23.214508821 +0000 2022-02-21T17:06:24.930 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-02-21 17:06:23.214508821 +0000 2022-02-21T17:06:24.930 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-02-21 17:06:23.214508821 +0000 2022-02-21T17:06:24.930 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-02-21T17:06:24.937 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-02-21T17:06:24.938 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-02-21T17:06:24.938 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000160122 s, 3.2 MB/s 2022-02-21T17:06:24.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-02-21T17:06:25.025 INFO:teuthology.orchestra.run.smithi005.stdout:loop 2022-02-21T17:06:25.026 INFO:tasks.nvme_loop:Connecting nvme_loop smithi005:/dev/vg_nvme/lv_1... 2022-02-21T17:06:25.026 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-02-21T17:06:25.049 INFO:teuthology.orchestra.run.smithi005.stdout:1 2022-02-21T17:06:25.075 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_11 2022-02-21T17:06:25.093 INFO:tasks.nvme_loop:Connecting nvme_loop smithi005:/dev/vg_nvme/lv_2... 2022-02-21T17:06:25.094 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-02-21T17:06:25.152 INFO:teuthology.orchestra.run.smithi005.stdout:1 2022-02-21T17:06:25.176 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_21 2022-02-21T17:06:25.197 INFO:tasks.nvme_loop:Connecting nvme_loop smithi005:/dev/vg_nvme/lv_3... 2022-02-21T17:06:25.197 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-02-21T17:06:25.220 INFO:teuthology.orchestra.run.smithi005.stdout:1 2022-02-21T17:06:25.245 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_31 2022-02-21T17:06:25.271 INFO:tasks.nvme_loop:Connecting nvme_loop smithi005:/dev/vg_nvme/lv_4... 2022-02-21T17:06:25.271 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-02-21T17:06:25.290 INFO:teuthology.orchestra.run.smithi005.stdout:1 2022-02-21T17:06:25.314 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_41 2022-02-21T17:06:25.336 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:06:25.337 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:06:25.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo nvme list 2022-02-21T17:06:25.399 INFO:teuthology.orchestra.run.smithi005.stdout:Node SN Model Namespace Usage Format FW Rev 2022-02-21T17:06:25.400 INFO:teuthology.orchestra.run.smithi005.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-02-21T17:06:25.400 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme0n1 PHMB751300EW480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-02-21T17:06:25.400 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme1n1 ecbef28d6af4c047 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:25.401 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme2n1 81b825b9d5b9d5d8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:25.401 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme3n1 b9d3b839572c63dd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:25.401 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme4n1 67f0252fdf5cf388 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:25.401 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-02-21T17:06:25.402 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:06:25.402 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/scratch_devs 2022-02-21T17:06:25.416 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:06:25.416 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:06:25.424 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T17:06:25.425 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-02-21T17:06:25.474 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-02-21T17:06:25.474 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:25.475 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-02-21T17:06:25.475 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:25.475 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2022-02-21 17:06:23.299624273 +0000 2022-02-21T17:06:25.475 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2022-02-21 17:06:23.299624273 +0000 2022-02-21T17:06:25.476 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2022-02-21 17:06:23.299624273 +0000 2022-02-21T17:06:25.476 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2022-02-21T17:06:25.480 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2022-02-21T17:06:25.481 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2022-02-21T17:06:25.481 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000533143 s, 960 kB/s 2022-02-21T17:06:25.485 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-02-21T17:06:25.490 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-02-21T17:06:25.491 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:25.491 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-02-21T17:06:25.491 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:25.491 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2022-02-21 17:06:23.331624274 +0000 2022-02-21T17:06:25.492 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2022-02-21 17:06:23.331624274 +0000 2022-02-21T17:06:25.492 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2022-02-21 17:06:23.331624274 +0000 2022-02-21T17:06:25.492 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2022-02-21T17:06:25.500 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2022-02-21T17:06:25.501 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2022-02-21T17:06:25.501 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000480427 s, 1.1 MB/s 2022-02-21T17:06:25.505 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-02-21T17:06:25.511 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-02-21T17:06:25.512 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:25.512 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-02-21T17:06:25.512 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:25.512 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2022-02-21 17:06:23.363624275 +0000 2022-02-21T17:06:25.513 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2022-02-21 17:06:23.363624275 +0000 2022-02-21T17:06:25.513 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2022-02-21 17:06:23.363624275 +0000 2022-02-21T17:06:25.513 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2022-02-21T17:06:25.520 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2022-02-21T17:06:25.520 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2022-02-21T17:06:25.521 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000501 s, 1.0 MB/s 2022-02-21T17:06:25.524 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-02-21T17:06:25.530 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-02-21T17:06:25.531 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:25.531 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-02-21T17:06:25.531 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:25.532 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2022-02-21 17:06:23.387624276 +0000 2022-02-21T17:06:25.532 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2022-02-21 17:06:23.387624276 +0000 2022-02-21T17:06:25.532 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2022-02-21 17:06:23.387624276 +0000 2022-02-21T17:06:25.532 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2022-02-21T17:06:25.539 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2022-02-21T17:06:25.539 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2022-02-21T17:06:25.540 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000555804 s, 921 kB/s 2022-02-21T17:06:25.542 DEBUG:teuthology.orchestra.run.smithi033:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-02-21T17:06:25.631 INFO:teuthology.orchestra.run.smithi033.stdout:loop 2022-02-21T17:06:25.632 INFO:tasks.nvme_loop:Connecting nvme_loop smithi033:/dev/vg_nvme/lv_1... 2022-02-21T17:06:25.633 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-02-21T17:06:25.651 INFO:teuthology.orchestra.run.smithi033.stdout:1 2022-02-21T17:06:25.677 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/vg_nvme/lv_11 2022-02-21T17:06:25.695 INFO:tasks.nvme_loop:Connecting nvme_loop smithi033:/dev/vg_nvme/lv_2... 2022-02-21T17:06:25.696 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-02-21T17:06:25.713 INFO:teuthology.orchestra.run.smithi033.stdout:1 2022-02-21T17:06:25.735 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/vg_nvme/lv_21 2022-02-21T17:06:25.755 INFO:tasks.nvme_loop:Connecting nvme_loop smithi033:/dev/vg_nvme/lv_3... 2022-02-21T17:06:25.756 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-02-21T17:06:25.772 INFO:teuthology.orchestra.run.smithi033.stdout:1 2022-02-21T17:06:25.791 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/vg_nvme/lv_31 2022-02-21T17:06:25.815 INFO:tasks.nvme_loop:Connecting nvme_loop smithi033:/dev/vg_nvme/lv_4... 2022-02-21T17:06:25.816 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-02-21T17:06:25.882 INFO:teuthology.orchestra.run.smithi033.stdout:1 2022-02-21T17:06:25.905 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/vg_nvme/lv_41 2022-02-21T17:06:25.926 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:06:25.926 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:06:25.933 DEBUG:teuthology.orchestra.run.smithi033:> sudo nvme list 2022-02-21T17:06:25.943 INFO:teuthology.orchestra.run.smithi033.stdout:Node SN Model Namespace Usage Format FW Rev 2022-02-21T17:06:25.943 INFO:teuthology.orchestra.run.smithi033.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-02-21T17:06:25.944 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme0n1 CVFT5330008Q400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-02-21T17:06:25.944 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme1n1 5d7adad5ce594450 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:25.944 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme2n1 65cdb9f2085dcfc1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:25.945 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme3n1 c3005aa223713cd8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:25.946 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-02-21T17:06:26.948 DEBUG:teuthology.orchestra.run.smithi033:> sudo nvme list 2022-02-21T17:06:26.962 INFO:teuthology.orchestra.run.smithi033.stdout:Node SN Model Namespace Usage Format FW Rev 2022-02-21T17:06:26.962 INFO:teuthology.orchestra.run.smithi033.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-02-21T17:06:26.963 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme0n1 CVFT5330008Q400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-02-21T17:06:26.963 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme1n1 5d7adad5ce594450 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:26.963 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme2n1 65cdb9f2085dcfc1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:26.964 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme3n1 c3005aa223713cd8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:26.964 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme4n1 25d0bc35ace1f95e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:26.966 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-02-21T17:06:26.967 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:06:26.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/scratch_devs 2022-02-21T17:06:27.019 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:06:27.020 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:06:27.027 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-02-21T17:06:27.028 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-02-21T17:06:27.073 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-02-21T17:06:27.074 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:27.074 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 498 Links: 1 2022-02-21T17:06:27.074 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:27.074 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2022-02-21 17:06:23.424410298 +0000 2022-02-21T17:06:27.074 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2022-02-21 17:06:23.424410298 +0000 2022-02-21T17:06:27.075 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2022-02-21 17:06:23.424410298 +0000 2022-02-21T17:06:27.075 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2022-02-21T17:06:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2022-02-21T17:06:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2022-02-21T17:06:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000595352 s, 860 kB/s 2022-02-21T17:06:27.087 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-02-21T17:06:27.092 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-02-21T17:06:27.093 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:27.093 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 504 Links: 1 2022-02-21T17:06:27.093 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:27.093 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2022-02-21 17:06:23.456410297 +0000 2022-02-21T17:06:27.094 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2022-02-21 17:06:23.456410297 +0000 2022-02-21T17:06:27.094 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2022-02-21 17:06:23.456410297 +0000 2022-02-21T17:06:27.094 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2022-02-21T17:06:27.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2022-02-21T17:06:27.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2022-02-21T17:06:27.100 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000483583 s, 1.1 MB/s 2022-02-21T17:06:27.104 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-02-21T17:06:27.111 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-02-21T17:06:27.111 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:27.112 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-02-21T17:06:27.112 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:27.113 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2022-02-21 17:06:23.532410295 +0000 2022-02-21T17:06:27.113 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2022-02-21 17:06:23.532410295 +0000 2022-02-21T17:06:27.113 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2022-02-21 17:06:23.532410295 +0000 2022-02-21T17:06:27.114 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2022-02-21T17:06:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2022-02-21T17:06:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2022-02-21T17:06:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000461294 s, 1.1 MB/s 2022-02-21T17:06:27.124 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-02-21T17:06:27.177 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-02-21T17:06:27.178 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-02-21T17:06:27.178 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-02-21T17:06:27.179 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-02-21T17:06:27.179 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2022-02-21 17:06:23.560410295 +0000 2022-02-21T17:06:27.180 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2022-02-21 17:06:23.560410295 +0000 2022-02-21T17:06:27.180 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2022-02-21 17:06:23.560410295 +0000 2022-02-21T17:06:27.180 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2022-02-21T17:06:27.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2022-02-21T17:06:27.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2022-02-21T17:06:27.187 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.00057735 s, 887 kB/s 2022-02-21T17:06:27.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-02-21T17:06:27.304 INFO:teuthology.orchestra.run.smithi067.stdout:loop 2022-02-21T17:06:27.307 INFO:tasks.nvme_loop:Connecting nvme_loop smithi067:/dev/vg_nvme/lv_1... 2022-02-21T17:06:27.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-02-21T17:06:27.324 INFO:teuthology.orchestra.run.smithi067.stdout:1 2022-02-21T17:06:27.342 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/vg_nvme/lv_11 2022-02-21T17:06:27.362 INFO:tasks.nvme_loop:Connecting nvme_loop smithi067:/dev/vg_nvme/lv_2... 2022-02-21T17:06:27.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-02-21T17:06:27.429 INFO:teuthology.orchestra.run.smithi067.stdout:1 2022-02-21T17:06:27.448 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/vg_nvme/lv_21 2022-02-21T17:06:27.469 INFO:tasks.nvme_loop:Connecting nvme_loop smithi067:/dev/vg_nvme/lv_3... 2022-02-21T17:06:27.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-02-21T17:06:27.489 INFO:teuthology.orchestra.run.smithi067.stdout:1 2022-02-21T17:06:27.512 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/vg_nvme/lv_31 2022-02-21T17:06:27.530 INFO:tasks.nvme_loop:Connecting nvme_loop smithi067:/dev/vg_nvme/lv_4... 2022-02-21T17:06:27.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-02-21T17:06:27.548 INFO:teuthology.orchestra.run.smithi067.stdout:1 2022-02-21T17:06:27.577 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/vg_nvme/lv_41 2022-02-21T17:06:27.595 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:06:27.595 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:06:27.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo nvme list 2022-02-21T17:06:27.657 INFO:teuthology.orchestra.run.smithi067.stdout:Node SN Model Namespace Usage Format FW Rev 2022-02-21T17:06:27.657 INFO:teuthology.orchestra.run.smithi067.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-02-21T17:06:27.657 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/nvme0n1 CVFT53440042400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-02-21T17:06:27.658 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/nvme1n1 08403f93955d8118 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:27.658 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/nvme2n1 8626d5856b483e86 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:27.658 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/nvme3n1 c31a893fbe690ec8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:27.658 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/nvme4n1 27815322e1bb016f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-02-21T17:06:27.659 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-02-21T17:06:27.660 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:06:27.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/scratch_devs 2022-02-21T17:06:27.672 INFO:teuthology.run_tasks:Running task rook... 2022-02-21T17:06:27.747 INFO:tasks.rook:Rook start 2022-02-21T17:06:27.747 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b05010b68efc838fdc35b41106e2ebdabe395a3e', 'rook_image': 'rook/ceph:master'} 2022-02-21T17:06:27.748 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e 2022-02-21T17:06:27.748 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-02-21T17:06:27.748 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-02-21T17:06:27.760 INFO:teuthology.orchestra.run.smithi005.stderr:Cloning into 'rook'... 2022-02-21T17:06:33.818 INFO:tasks.rook:/home/teuthworker 2022-02-21T17:06:33.818 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/sbin:/usr/sbin ip addr show 2022-02-21T17:06:33.823 INFO:teuthology.orchestra.run.smithi005.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-02-21T17:06:33.824 INFO:teuthology.orchestra.run.smithi005.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-02-21T17:06:33.824 INFO:teuthology.orchestra.run.smithi005.stdout: inet 127.0.0.1/8 scope host lo 2022-02-21T17:06:33.824 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2022-02-21T17:06:33.826 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 ::1/128 scope host 2022-02-21T17:06:33.826 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2022-02-21T17:06:33.826 INFO:teuthology.orchestra.run.smithi005.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-02-21T17:06:33.826 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:6b:fd:de brd ff:ff:ff:ff:ff:ff 2022-02-21T17:06:33.827 INFO:teuthology.orchestra.run.smithi005.stdout: altname enp7s0f0 2022-02-21T17:06:33.827 INFO:teuthology.orchestra.run.smithi005.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-02-21T17:06:33.827 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:6b:fd:df brd ff:ff:ff:ff:ff:ff 2022-02-21T17:06:33.827 INFO:teuthology.orchestra.run.smithi005.stdout: altname enp7s0f1 2022-02-21T17:06:33.827 INFO:teuthology.orchestra.run.smithi005.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-02-21T17:06:33.828 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:bd:14:42 brd ff:ff:ff:ff:ff:ff 2022-02-21T17:06:33.828 INFO:teuthology.orchestra.run.smithi005.stdout: altname enp1s0f0 2022-02-21T17:06:33.828 INFO:teuthology.orchestra.run.smithi005.stdout: inet 172.21.15.5/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-02-21T17:06:33.828 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft 42854sec preferred_lft 42854sec 2022-02-21T17:06:33.829 INFO:teuthology.orchestra.run.smithi005.stdout: inet6 fe80::ec4:7aff:febd:1442/64 scope link 2022-02-21T17:06:33.829 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2022-02-21T17:06:33.829 INFO:teuthology.orchestra.run.smithi005.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-02-21T17:06:33.829 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 0c:c4:7a:bd:14:43 brd ff:ff:ff:ff:ff:ff 2022-02-21T17:06:33.830 INFO:teuthology.orchestra.run.smithi005.stdout: altname enp1s0f1 2022-02-21T17:06:33.830 INFO:teuthology.orchestra.run.smithi005.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-02-21T17:06:33.830 INFO:teuthology.orchestra.run.smithi005.stdout: link/ether 02:42:2e:ae:55:68 brd ff:ff:ff:ff:ff:ff 2022-02-21T17:06:33.830 INFO:teuthology.orchestra.run.smithi005.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-02-21T17:06:33.830 INFO:teuthology.orchestra.run.smithi005.stdout: valid_lft forever preferred_lft forever 2022-02-21T17:06:33.832 DEBUG:teuthology.orchestra.run.smithi005:> which systemctl 2022-02-21T17:06:33.871 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/systemctl 2022-02-21T17:06:33.872 DEBUG:teuthology.orchestra.run.smithi005:> true 2022-02-21T17:06:33.902 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2022-02-21T17:06:33.952 INFO:teuthology.orchestra.run.smithi005.stdout:Ubuntu 2022-02-21T17:06:33.958 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-02-21T17:06:33.958 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-02-21T17:06:34.008 DEBUG:teuthology.orchestra.remote:smithi005:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T17:06:34.022 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:06:34.023 DEBUG:teuthology.orchestra.run.smithi005:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-02-21T17:06:34.029 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-02-21T17:06:34.074 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:06:34.075 DEBUG:teuthology.orchestra.run.smithi005:> dd of=operator.yaml 2022-02-21T17:06:34.082 INFO:tasks.rook:Deploying operator 2022-02-21T17:06:34.137 DEBUG:teuthology.orchestra.remote:smithi005:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T17:06:34.164 DEBUG:teuthology.orchestra.remote:smithi005:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T17:06:34.179 DEBUG:teuthology.orchestra.run.smithi005:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-02-21T17:06:34.542 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-02-21T17:06:34.553 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-02-21T17:06:34.580 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-02-21T17:06:34.605 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-02-21T17:06:34.713 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-02-21T17:06:34.772 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-02-21T17:06:34.903 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-02-21T17:06:35.272 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-02-21T17:06:35.320 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-02-21T17:06:35.369 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-02-21T17:06:35.468 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-02-21T17:06:35.475 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-02-21T17:06:35.501 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-02-21T17:06:35.530 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-02-21T17:06:35.554 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-02-21T17:06:35.603 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-02-21T17:06:35.618 INFO:teuthology.orchestra.run.smithi005.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-02-21T17:06:35.676 INFO:teuthology.orchestra.run.smithi005.stdout:namespace/rook-ceph created 2022-02-21T17:06:35.700 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-02-21T17:06:35.785 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-02-21T17:06:35.790 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-02-21T17:06:35.802 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-02-21T17:06:36.040 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-02-21T17:06:36.069 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-02-21T17:06:36.111 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-02-21T17:06:36.203 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-02-21T17:06:36.206 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-02-21T17:06:36.265 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-02-21T17:06:36.497 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-02-21T17:06:36.511 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-02-21T17:06:36.564 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-02-21T17:06:36.587 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-02-21T17:06:36.645 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-02-21T17:06:36.649 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-02-21T17:06:36.660 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-02-21T17:06:36.707 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-02-21T17:06:36.738 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-02-21T17:06:36.802 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-02-21T17:06:36.806 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-02-21T17:06:36.839 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-02-21T17:06:36.843 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-02-21T17:06:36.852 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-02-21T17:06:36.860 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-02-21T17:06:36.869 INFO:teuthology.orchestra.run.smithi005.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-02-21T17:06:36.876 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-02-21T17:06:36.877 INFO:teuthology.orchestra.run.smithi005.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-02-21T17:06:36.965 INFO:teuthology.orchestra.run.smithi005.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-02-21T17:06:36.968 INFO:teuthology.orchestra.run.smithi005.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-02-21T17:06:36.986 INFO:teuthology.orchestra.run.smithi005.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-02-21T17:06:37.006 INFO:teuthology.orchestra.run.smithi005.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-02-21T17:06:37.012 INFO:teuthology.orchestra.run.smithi005.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-02-21T17:06:37.018 INFO:teuthology.orchestra.run.smithi005.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-02-21T17:06:37.046 INFO:teuthology.orchestra.run.smithi005.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-02-21T17:06:37.050 INFO:teuthology.orchestra.run.smithi005.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-02-21T17:06:37.060 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-02-21T17:06:37.094 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-02-21T17:06:37.098 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-02-21T17:06:37.114 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-02-21T17:06:37.127 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-02-21T17:06:37.135 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-02-21T17:06:37.143 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-02-21T17:06:37.152 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-02-21T17:06:37.160 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-02-21T17:06:37.182 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-02-21T17:06:37.193 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-02-21T17:06:37.244 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-02-21T17:06:37.248 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-02-21T17:06:37.251 INFO:teuthology.orchestra.run.smithi005.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-02-21T17:06:37.260 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-02-21T17:06:37.268 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/rook-ceph-mgr created 2022-02-21T17:06:37.290 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/rook-ceph-osd created 2022-02-21T17:06:37.302 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/rook-ceph-purge-osd created 2022-02-21T17:06:37.315 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/rook-ceph-system created 2022-02-21T17:06:37.332 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-02-21T17:06:37.391 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-02-21T17:06:37.402 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-02-21T17:06:37.415 INFO:teuthology.orchestra.run.smithi005.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-02-21T17:06:37.432 INFO:teuthology.orchestra.run.smithi005.stdout:configmap/rook-ceph-operator-config created 2022-02-21T17:06:37.453 INFO:teuthology.orchestra.run.smithi005.stdout:deployment.apps/rook-ceph-operator created 2022-02-21T17:06:37.457 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T17:06:37.542 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:06:37.543 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-operator-75dd789779-q2rkd 0/1 Pending 0 0s 2022-02-21T17:06:47.546 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T17:06:47.619 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:06:47.620 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-operator-75dd789779-q2rkd 0/1 ContainerCreating 0 10s 2022-02-21T17:06:57.623 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T17:06:57.688 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:06:57.688 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-operator-75dd789779-q2rkd 0/1 ContainerCreating 0 20s 2022-02-21T17:07:07.692 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T17:07:07.803 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:07.803 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-operator-75dd789779-q2rkd 0/1 ContainerCreating 0 30s 2022-02-21T17:07:17.807 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-02-21T17:07:17.876 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:17.876 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-operator-75dd789779-q2rkd 1/1 Running 0 40s 2022-02-21T17:07:17.878 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-q2rkd 2022-02-21T17:07:17.886 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:07:17.887 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:07:17.940 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:07:17.941 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:07:17.949 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:07:17.950 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2022-02-21T17:07:17.961 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-02-21T17:07:17.961 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-02-21T17:07:17.962 INFO:tasks.rook: override: [mon] debug mon = 20 2022-02-21T17:07:17.962 INFO:tasks.rook: override: [mon] debug ms = 1 2022-02-21T17:07:17.962 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-02-21T17:07:17.963 INFO:tasks.rook: override: [osd] debug ms = 1 2022-02-21T17:07:17.963 INFO:tasks.rook: override: [osd] debug osd = 20 2022-02-21T17:07:17.963 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-02-21T17:07:17.966 DEBUG:teuthology.orchestra.run.smithi005:> kubectl create -f - 2022-02-21T17:07:17.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:07.937476 I | rookcmd: starting Rook v1.8.0-alpha.0.326.gd73093883 with arguments '/usr/local/bin/rook ceph operator' 2022-02-21T17:07:17.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:07.937573 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-02-21T17:07:17.980 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:07.937578 I | cephcmd: starting Rook-Ceph operator 2022-02-21T17:07:17.980 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.103224 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-02-21T17:07:17.980 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.111513 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T17:07:17.980 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.111535 I | operator: watching all namespaces for Ceph CRs 2022-02-21T17:07:17.981 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.111600 I | operator: setting up schemes 2022-02-21T17:07:17.981 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.114890 I | operator: setting up the controller-runtime manager 2022-02-21T17:07:17.981 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.320282 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T17:07:17.981 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.339925 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T17:07:17.981 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.339982 I | ceph-cluster-controller: successfully started 2022-02-21T17:07:17.982 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340082 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T17:07:17.982 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340112 I | ceph-crashcollector-controller: successfully started 2022-02-21T17:07:17.982 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340142 I | ceph-block-pool-controller: successfully started 2022-02-21T17:07:17.982 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340215 I | ceph-object-store-user-controller: successfully started 2022-02-21T17:07:17.982 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340265 I | ceph-object-realm-controller: successfully started 2022-02-21T17:07:17.983 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340291 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T17:07:17.983 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340312 I | ceph-object-zone-controller: successfully started 2022-02-21T17:07:17.983 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340505 I | ceph-object-controller: successfully started 2022-02-21T17:07:17.983 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340563 I | ceph-file-controller: successfully started 2022-02-21T17:07:17.983 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340624 I | ceph-nfs-controller: successfully started 2022-02-21T17:07:17.984 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340692 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T17:07:17.984 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340731 I | ceph-client-controller: successfully started 2022-02-21T17:07:17.984 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340771 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T17:07:17.984 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340812 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T17:07:17.984 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340844 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T17:07:17.985 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340880 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T17:07:17.985 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340933 I | ceph-bucket-topic: successfully started 2022-02-21T17:07:17.985 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340961 I | ceph-bucket-notification: successfully started 2022-02-21T17:07:17.985 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340979 I | ceph-bucket-notification: successfully started 2022-02-21T17:07:17.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.340999 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T17:07:17.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.342649 I | operator: starting the controller-runtime manager 2022-02-21T17:07:17.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.447289 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T17:07:17.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.447344 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T17:07:17.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.447365 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T17:07:17.987 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.451164 I | ceph-csi: CSI Ceph RBD driver disabled 2022-02-21T17:07:17.987 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.451208 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-02-21T17:07:17.987 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.451394 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T17:07:17.987 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.453751 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-02-21T17:07:17.987 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.460953 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-02-21T17:07:17.988 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.460977 I | ceph-csi: CSI CephFS driver disabled 2022-02-21T17:07:17.988 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.460992 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-02-21T17:07:17.988 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.463409 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-02-21T17:07:17.988 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:08.470286 I | ceph-csi: successfully removed CSI CephFS driver 2022-02-21T17:07:18.841 INFO:teuthology.orchestra.run.smithi005.stdout:configmap/rook-config-override created 2022-02-21T17:07:18.846 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-02-21T17:07:18.846 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:07:18.847 DEBUG:teuthology.orchestra.run.smithi005:> dd of=cluster.yaml 2022-02-21T17:07:18.852 DEBUG:teuthology.orchestra.run.smithi005:> kubectl create -f cluster.yaml 2022-02-21T17:07:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-02-21T17:07:20.980 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:20.975574 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-02-21T17:07:20.986 DEBUG:teuthology.orchestra.remote:smithi005:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T17:07:21.001 DEBUG:teuthology.orchestra.run.smithi005:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-02-21T17:07:21.027 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.022812 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T17:07:21.032 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.028155 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-02-21T17:07:21.039 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.034480 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-02-21T17:07:21.051 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.047097 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T17:07:21.081 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075097 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-02-21T17:07:21.081 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075122 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-02-21T17:07:21.082 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075131 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-02-21T17:07:21.082 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075138 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-02-21T17:07:21.082 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075146 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-02-21T17:07:21.083 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075153 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-02-21T17:07:21.083 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075161 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-02-21T17:07:21.083 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075169 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-02-21T17:07:21.083 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075189 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-02-21T17:07:21.083 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075199 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-02-21T17:07:21.084 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075206 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-02-21T17:07:21.084 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075214 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-02-21T17:07:21.084 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075222 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-02-21T17:07:21.084 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075228 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-02-21T17:07:21.084 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075234 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-02-21T17:07:21.085 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075242 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-02-21T17:07:21.085 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075349 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:07:21.085 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:21.075379 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:07:21.268 INFO:teuthology.orchestra.run.smithi005.stdout:deployment.apps/rook-ceph-tools created 2022-02-21T17:07:21.275 INFO:tasks.rook:Waiting for tools container to start 2022-02-21T17:07:21.275 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:07:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 Pending 0 0s 2022-02-21T17:07:26.406 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:07:26.473 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:26.473 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 5s 2022-02-21T17:07:31.476 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:07:31.541 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:31.541 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 10s 2022-02-21T17:07:34.599 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590766 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-02-21T17:07:34.599 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590797 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-02-21T17:07:34.599 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590804 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T17:07:34.599 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590808 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T17:07:34.600 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590813 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T17:07:34.600 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590816 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T17:07:34.600 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590819 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T17:07:34.600 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590823 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T17:07:34.600 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590827 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T17:07:34.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590830 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T17:07:34.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590834 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T17:07:34.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590837 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T17:07:34.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590841 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-02-21T17:07:34.602 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590844 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-02-21T17:07:34.602 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590848 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-02-21T17:07:34.602 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590852 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-02-21T17:07:34.602 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590857 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-02-21T17:07:34.602 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590860 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-02-21T17:07:34.603 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590864 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-02-21T17:07:34.603 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590868 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T17:07:34.603 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590872 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T17:07:34.603 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590876 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-02-21T17:07:34.604 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590880 I | ceph-csi: Kubernetes version is 1.23 2022-02-21T17:07:34.604 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590884 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-02-21T17:07:34.604 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.590888 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-02-21T17:07:34.608 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.603764 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-02-21T17:07:34.617 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.612686 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:07:34.617 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.612713 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:07:34.618 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.612718 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:07:34.618 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.612721 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:07:34.618 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.612724 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:07:34.618 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.612727 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:07:34.618 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.612730 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-02-21T17:07:34.709 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.704535 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:07:34.709 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.704563 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:07:34.709 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.704571 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-02-21T17:07:34.850 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:34.846425 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-02-21T17:07:35.022 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.017930 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:07:35.023 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.017952 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:07:35.023 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.017956 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-02-21T17:07:35.042 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.037505 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:07:35.042 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.037531 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:07:35.042 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.037539 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-02-21T17:07:35.115 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.110784 I | ceph-csi: successfully started CSI CephFS driver 2022-02-21T17:07:35.256 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.251861 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-02-21T17:07:35.380 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.376230 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-02-21T17:07:35.381 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.376260 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-02-21T17:07:35.392 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:07:35.387541 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-02-21T17:07:36.544 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:07:36.643 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:36.643 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 15s 2022-02-21T17:07:41.646 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:07:41.712 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:41.713 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 20s 2022-02-21T17:07:46.715 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:07:46.784 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:46.785 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 25s 2022-02-21T17:07:51.788 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:07:51.859 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:51.859 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 30s 2022-02-21T17:07:56.862 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:07:56.926 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:07:56.926 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 35s 2022-02-21T17:08:01.928 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:08:01.994 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:08:01.994 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 40s 2022-02-21T17:08:05.894 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:05.884259 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T17:08:05.894 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:05.884285 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T17:08:05.894 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:05.884297 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T17:08:05.904 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:05.897250 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T17:08:06.034 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:06.026409 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-02-21 17:08:05.976265674 +0000 UTC m=+58.061319795 LastTransitionTime:2022-02-21 17:08:05.976265465 +0000 UTC m=+58.061319592}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T17:08:06.053 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:06.045689 I | op-mon: start running mons 2022-02-21T17:08:06.141 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:06.132992 I | op-mon: creating mon secrets for a new cluster 2022-02-21T17:08:06.162 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:06.154142 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-02-21T17:08:06.165 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:06.157463 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-02-21T17:08:06.899 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:06.891012 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:08:06.899 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:06.891172 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:08:06.996 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:08:07.060 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:08:07.061 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 46s 2022-02-21T17:08:08.474 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:08.466442 I | op-mon: targeting the mon count 3 2022-02-21T17:08:08.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:08.472559 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-02-21T17:08:08.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:08.492989 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-02-21T17:08:08.589 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:08.510252 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-02-21T17:08:09.626 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:09.617934 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi067 2022-02-21T17:08:09.626 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:09.617960 I | op-mon: mon a assigned to node smithi067 2022-02-21T17:08:09.628 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:09.618284 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi033 2022-02-21T17:08:09.628 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:09.618313 I | op-mon: mon b assigned to node smithi033 2022-02-21T17:08:09.941 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:09.934017 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi005 2022-02-21T17:08:09.942 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:09.934041 I | op-mon: mon c assigned to node smithi005 2022-02-21T17:08:10.298 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:10.290310 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-02-21T17:08:10.303 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:10.295354 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-02-21T17:08:10.820 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:10.812724 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-02-21T17:08:11.236 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:11.225537 I | op-mon: creating mon a 2022-02-21T17:08:11.717 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:11.707972 I | op-mon: parsing mon endpoints: 2022-02-21T17:08:11.718 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:11.708007 W | op-mon: ignoring invalid monitor 2022-02-21T17:08:11.718 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:11.708059 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-02-21T17:08:11.718 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:11.708079 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-02-21T17:08:11.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:11.710222 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-02-21T17:08:11.719 INFO:tasks.rook.operator.smithi005.stdout:I0221 17:08:11.710335 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-02-21T17:08:12.064 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:08:12.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:12.151270 I | op-mon: mon "a" endpoint is [v2:10.110.173.53:3300,v1:10.110.173.53:6789] 2022-02-21T17:08:12.612 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:08:12.613 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 51s 2022-02-21T17:08:14.153 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:14.145079 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-02-21T17:08:14.154 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:14.145148 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-02-21T17:08:14.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:14.148065 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789"]}] data:a=10.110.173.53:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:-1] 2022-02-21T17:08:14.375 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:14.367140 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:08:14.417 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:14.409977 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:08:14.710 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:14.702189 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-02-21T17:08:14.749 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:14.741336 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-02-21T17:08:15.080 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:15.072374 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789"]}] data:a=10.110.173.53:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:0] 2022-02-21T17:08:15.080 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:15.072414 I | op-mon: waiting for mon quorum with [a] 2022-02-21T17:08:15.086 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:15.079335 I | op-mon: mons running: [a] 2022-02-21T17:08:17.615 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:08:17.690 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:08:17.690 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 56s 2022-02-21T17:08:22.693 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:08:22.763 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:08:22.763 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 61s 2022-02-21T17:08:27.765 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:08:27.837 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:08:27.837 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 0/1 ContainerCreating 0 66s 2022-02-21T17:08:32.839 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-02-21T17:08:32.916 INFO:teuthology.orchestra.run.smithi005.stdout:NAME READY STATUS RESTARTS AGE 2022-02-21T17:08:32.916 INFO:teuthology.orchestra.run.smithi005.stdout:rook-ceph-tools-d6d7c985c-kqz7x 1/1 Running 0 71s 2022-02-21T17:08:32.918 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-02-21T17:08:32.919 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:08:35.219 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:35.211421 I | op-mon: mons running: [a] 2022-02-21T17:08:35.428 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:35.421091 I | op-mon: Monitors in quorum: [a] 2022-02-21T17:08:35.429 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:35.421119 I | op-mon: mons created: 1 2022-02-21T17:08:35.638 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:35.630861 I | op-mon: waiting for mon quorum with [a] 2022-02-21T17:08:35.665 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:35.657848 I | op-mon: mons running: [a] 2022-02-21T17:08:35.666 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:08:35.667 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:08:35.667 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:08:35.667 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:08:35.668 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:08:35.668 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:08:35.668 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:08:35.668 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:08:35.669 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:08:35.669 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:08:35.669 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:08:35.669 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:08:35.670 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:08:35.671 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:08:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:08:35.890 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:35.882251 I | op-mon: Monitors in quorum: [a] 2022-02-21T17:08:35.891 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:35.882292 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:08:36.168 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:36.160582 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:08:36.168 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:36.160614 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:08:36.467 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:36.459934 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:08:36.468 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:36.459978 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:08:36.751 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:36.743164 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:08:36.751 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:36.743195 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:08:37.019 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.012116 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:08:37.020 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.012150 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T17:08:37.246 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.238633 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T17:08:37.246 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.238662 I | op-mon: creating mon b 2022-02-21T17:08:37.437 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.429568 I | op-mon: mon "a" endpoint is [v2:10.110.173.53:3300,v1:10.110.173.53:6789] 2022-02-21T17:08:37.469 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.461423 I | op-mon: mon "b" endpoint is [v2:10.106.73.178:3300,v1:10.106.73.178:6789] 2022-02-21T17:08:37.553 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.541802 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789","10.106.73.178:6789"]}] data:a=10.110.173.53:6789,b=10.106.73.178:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:0] 2022-02-21T17:08:37.616 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.608877 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:08:37.617 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.609107 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:08:37.882 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.872448 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T17:08:37.889 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.881528 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T17:08:37.889 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:37.881559 I | op-mon: waiting for mon quorum with [a b] 2022-02-21T17:08:38.261 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:38.253429 I | op-mon: mon b is not yet running 2022-02-21T17:08:38.261 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:38.253468 I | op-mon: mons running: [a] 2022-02-21T17:08:38.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:38.488168 I | op-mon: Monitors in quorum: [a] 2022-02-21T17:08:38.543 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:38.535756 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-02-21T17:08:39.252 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:39.244693 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789","10.106.73.178:6789"]}] data:a=10.110.173.53:6789,b=10.106.73.178:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:1] 2022-02-21T17:08:39.252 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:39.244728 I | op-mon: waiting for mon quorum with [a b] 2022-02-21T17:08:39.654 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:39.647028 I | op-mon: mon b is not yet running 2022-02-21T17:08:39.655 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:39.647064 I | op-mon: mons running: [a] 2022-02-21T17:08:39.904 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:39.896252 I | op-mon: Monitors in quorum: [a] 2022-02-21T17:08:39.904 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:39.896285 I | op-mon: mons created: 2 2022-02-21T17:08:40.119 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:40.111458 I | op-mon: waiting for mon quorum with [a b] 2022-02-21T17:08:40.134 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:40.126853 I | op-mon: mon b is not yet running 2022-02-21T17:08:40.134 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:40.126876 I | op-mon: mons running: [a] 2022-02-21T17:08:45.247 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:45.239454 I | op-mon: mon b is not yet running 2022-02-21T17:08:45.247 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:45.239487 I | op-mon: mons running: [a] 2022-02-21T17:08:45.675 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:08:46.088 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:08:46.089 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:08:46.089 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:08:46.089 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:08:46.089 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:08:46.090 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:08:46.090 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:08:46.090 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:08:46.090 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:08:46.091 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:08:46.091 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:08:46.091 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:08:46.091 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:08:46.093 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:08:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:08:50.600 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:50.593329 I | op-mon: mon b is not yet running 2022-02-21T17:08:50.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:50.593359 I | op-mon: mons running: [a] 2022-02-21T17:08:55.687 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:55.679887 I | op-mon: mon b is not yet running 2022-02-21T17:08:55.687 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:08:55.679918 I | op-mon: mons running: [a] 2022-02-21T17:08:56.096 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:08:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:08:56.578 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:08:56.578 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:08:56.578 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:08:56.578 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:08:56.579 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:08:56.579 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:08:56.579 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:08:56.579 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:08:56.580 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:08:56.580 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:08:56.580 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:08:56.581 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:08:56.582 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:08:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:09:00.736 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:00.729304 I | op-mon: mon b is not yet running 2022-02-21T17:09:00.737 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:00.729341 I | op-mon: mons running: [a] 2022-02-21T17:09:05.873 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:05.866007 I | op-mon: mons running: [a b] 2022-02-21T17:09:06.106 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:06.099184 W | op-mon: failed to find initial monitor b in mon map 2022-02-21T17:09:06.106 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:06.099214 W | op-mon: monitor b is not in quorum list 2022-02-21T17:09:06.586 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:09:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:09:07.084 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:09:07.084 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:09:07.085 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:09:07.085 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:09:07.085 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:09:07.085 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:09:07.086 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:09:07.086 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:09:07.086 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:09:07.087 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:09:07.087 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:09:07.087 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:09:07.089 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:09:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:09:11.123 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:11.116250 I | op-mon: mons running: [a b] 2022-02-21T17:09:11.355 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:11.347870 W | op-mon: failed to find initial monitor b in mon map 2022-02-21T17:09:11.355 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:11.347902 W | op-mon: monitor b is not in quorum list 2022-02-21T17:09:16.369 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:16.362581 I | op-mon: mons running: [a b] 2022-02-21T17:09:16.619 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:16.611915 W | op-mon: failed to find initial monitor b in mon map 2022-02-21T17:09:16.619 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:16.611947 W | op-mon: monitor b is not in quorum list 2022-02-21T17:09:17.093 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:09:17.593 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:09:17.593 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:09:17.594 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:09:17.594 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:09:17.594 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:09:17.594 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:09:17.595 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:09:17.595 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:09:17.595 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:09:17.595 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:09:17.596 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:09:17.596 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:09:17.596 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:09:17.598 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:09:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:09:21.634 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:21.627245 I | op-mon: mons running: [a b] 2022-02-21T17:09:21.875 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:21.868486 W | op-mon: failed to find initial monitor b in mon map 2022-02-21T17:09:21.875 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:21.868516 W | op-mon: monitor b is not in quorum list 2022-02-21T17:09:26.890 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:26.883646 I | op-mon: mons running: [a b] 2022-02-21T17:09:27.134 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:27.128009 W | op-mon: failed to find initial monitor b in mon map 2022-02-21T17:09:27.135 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:27.128042 W | op-mon: monitor b is not in quorum list 2022-02-21T17:09:27.603 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:09:28.070 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:09:28.071 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:09:28.071 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:09:28.071 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:09:28.072 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:09:28.072 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:09:28.073 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:09:28.073 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:09:28.073 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:09:28.074 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:09:28.074 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:09:28.074 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:09:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:09:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:09:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:09:32.165 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:32.159155 I | op-mon: mons running: [a b] 2022-02-21T17:09:32.401 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:32.394211 W | op-mon: failed to find initial monitor b in mon map 2022-02-21T17:09:32.401 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:32.394242 W | op-mon: monitor b is not in quorum list 2022-02-21T17:09:37.434 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:37.427964 I | op-mon: mon b is not yet running 2022-02-21T17:09:37.435 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:37.427998 I | op-mon: mons running: [a] 2022-02-21T17:09:38.079 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:09:38.554 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:09:38.555 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:09:38.555 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:09:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:09:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:09:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:09:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:09:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:09:38.557 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:09:38.557 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:09:38.557 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:09:38.557 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:09:38.557 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:09:38.559 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:09:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:09:42.452 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:42.445389 I | op-mon: mons running: [a b] 2022-02-21T17:09:44.096 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:44.090086 I | op-mon: Monitors in quorum: [a b] 2022-02-21T17:09:44.097 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:44.090120 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:09:44.328 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:44.322032 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:09:44.329 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:44.322075 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:09:44.565 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:44.558407 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:09:44.565 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:44.558438 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:09:44.804 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:44.797110 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:09:44.804 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:44.797142 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:09:45.066 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:45.059565 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:09:45.066 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:45.059610 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T17:09:45.279 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:45.272427 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T17:09:45.279 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:45.272461 I | op-mon: creating mon c 2022-02-21T17:09:45.390 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:45.383575 I | op-mon: mon "a" endpoint is [v2:10.110.173.53:3300,v1:10.110.173.53:6789] 2022-02-21T17:09:45.463 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:45.456747 I | op-mon: mon "b" endpoint is [v2:10.106.73.178:3300,v1:10.106.73.178:6789] 2022-02-21T17:09:45.536 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:45.529739 I | op-mon: mon "c" endpoint is [v2:10.109.91.196:3300,v1:10.109.91.196:6789] 2022-02-21T17:09:45.685 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:45.678575 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789","10.106.73.178:6789","10.109.91.196:6789"]}] data:c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:1] 2022-02-21T17:09:46.283 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:46.276918 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:09:46.286 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:46.277208 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:09:46.693 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:46.687143 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-02-21T17:09:46.711 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:46.704968 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T17:09:46.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:46.718056 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T17:09:46.731 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:46.725674 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-02-21T17:09:46.741 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:46.735167 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-02-21T17:09:46.892 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:46.886201 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-02-21T17:09:47.685 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:47.678340 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789","10.106.73.178:6789","10.109.91.196:6789"]}] data:a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:2] 2022-02-21T17:09:47.685 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:47.678383 I | op-mon: waiting for mon quorum with [a b c] 2022-02-21T17:09:48.286 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:48.279810 I | op-mon: mon c is not yet running 2022-02-21T17:09:48.287 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:48.279847 I | op-mon: mons running: [a b] 2022-02-21T17:09:48.518 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:48.511442 I | op-mon: Monitors in quorum: [a b] 2022-02-21T17:09:48.518 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:48.511473 I | op-mon: mons created: 3 2022-02-21T17:09:48.564 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:09:48.759 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:48.752905 I | op-mon: waiting for mon quorum with [a b c] 2022-02-21T17:09:48.885 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:48.878729 I | op-mon: mon c is not yet running 2022-02-21T17:09:48.885 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:48.878757 I | op-mon: mons running: [a b] 2022-02-21T17:09:49.039 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:09:49.039 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:09:49.040 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:09:49.040 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:09:49.040 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:09:49.040 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:09:49.041 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:09:49.041 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:09:49.041 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:09:49.042 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:09:49.042 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:09:49.042 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:09:49.042 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:09:49.049 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:09:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:09:53.905 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:53.899341 I | op-mon: mon c is not yet running 2022-02-21T17:09:53.906 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:53.899375 I | op-mon: mons running: [a b] 2022-02-21T17:09:58.927 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:58.921179 I | op-mon: mon c is not yet running 2022-02-21T17:09:58.928 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:09:58.921217 I | op-mon: mons running: [a b] 2022-02-21T17:09:59.053 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:09:59.541 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:09:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:09:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:09:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:09:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:09:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:09:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:09:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:09:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:09:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:09:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:09:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:09:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:09:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:09:59.548 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:10:03.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:03.941881 I | op-mon: mon c is not yet running 2022-02-21T17:10:03.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:03.941921 I | op-mon: mons running: [a b] 2022-02-21T17:10:09.270 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:09.264262 I | op-mon: mon c is not yet running 2022-02-21T17:10:09.270 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:09.264299 I | op-mon: mons running: [a b] 2022-02-21T17:10:09.553 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:10:10.059 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:10:10.060 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:10:10.060 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:10:10.060 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:10:10.061 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:10:10.061 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:10:10.061 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:10:10.061 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:10:10.062 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:10:10.062 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:10:10.062 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:10:10.062 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:10:10.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:10:10.064 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:10:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:10:14.580 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:14.574102 I | op-mon: mon c is not yet running 2022-02-21T17:10:14.580 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:14.574138 I | op-mon: mons running: [a b] 2022-02-21T17:10:19.969 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:19.963180 I | op-mon: mon c is not yet running 2022-02-21T17:10:19.969 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:19.963215 I | op-mon: mons running: [a b] 2022-02-21T17:10:20.069 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:10:20.570 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:10:20.571 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:10:20.571 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:10:20.572 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:10:20.572 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:10:20.572 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:10:20.572 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:10:20.573 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:10:20.573 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:10:20.573 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:10:20.573 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:10:20.573 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:10:20.574 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:10:20.576 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:10:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:10:24.997 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:24.989460 I | op-mon: mon c is not yet running 2022-02-21T17:10:24.998 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:24.989500 I | op-mon: mons running: [a b] 2022-02-21T17:10:30.022 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:30.016476 I | op-mon: mons running: [a b c] 2022-02-21T17:10:30.271 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:30.265352 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:10:30.271 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:30.265394 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:10:30.489 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:30.483286 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:10:30.489 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:30.483347 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:10:30.581 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:10:30.736 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:30.730718 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:10:30.737 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:30.730763 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:10:30.956 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:30.950730 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:10:30.957 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:30.950763 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:10:31.035 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:10:31.035 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:10:31.036 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:10:31.036 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:10:31.036 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:10:31.036 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:10:31.037 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:10:31.037 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:10:31.037 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:10:31.038 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:10:31.038 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:10:31.038 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:10:31.038 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:10:31.039 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:10:31.040 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:10:31.178 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:31.173266 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:10:31.179 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:31.173301 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T17:10:31.401 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:31.396212 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T17:10:31.402 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:31.396264 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-02-21T17:10:31.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:31.718616 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-02-21T17:10:32.050 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:32.044688 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-02-21T17:10:32.426 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:32.420948 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-02-21T17:10:32.848 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:32.842812 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-02-21T17:10:32.848 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:32.842847 I | cephclient: getting or creating ceph auth key "client.crash" 2022-02-21T17:10:33.168 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:33.162394 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-02-21T17:10:33.415 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:33.410276 I | cephclient: successfully enabled msgr2 protocol 2022-02-21T17:10:33.416 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:33.410317 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-02-21T17:10:33.416 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:33.410326 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-02-21T17:10:33.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:33.714058 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-02-21T17:10:33.741 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:33.735510 I | op-mgr: start running mgr 2022-02-21T17:10:33.742 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:33.735547 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-02-21T17:10:34.265 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:34.259961 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi033": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T17:10:34.377 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:34.372418 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi033": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T17:10:41.042 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:10:41.508 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:10:41.509 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2022-02-21T17:10:41.509 INFO:teuthology.orchestra.run.smithi005.stderr:pg stat 2022-02-21T17:10:41.510 INFO:teuthology.orchestra.run.smithi005.stderr:pg getmap 2022-02-21T17:10:41.510 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-02-21T17:10:41.510 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-02-21T17:10:41.510 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_pools_json 2022-02-21T17:10:41.511 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-pool [...] 2022-02-21T17:10:41.511 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-primary [] [...] 2022-02-21T17:10:41.511 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls-by-osd [] [...] 2022-02-21T17:10:41.511 INFO:teuthology.orchestra.run.smithi005.stderr:pg ls [] [...] 2022-02-21T17:10:41.511 INFO:teuthology.orchestra.run.smithi005.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-02-21T17:10:41.512 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2022-02-21T17:10:41.513 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 22 2022-02-21T17:10:41.515 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-02-21T17:10:51.518 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:10:52.350 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:10:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-02-21T17:10:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:command terminated with exit code 2 2022-02-21T17:10:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-02-21T17:10:55.172 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.167187 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-02-21T17:10:55.177 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.171765 I | op-mgr: setting services to point to mgr "a" 2022-02-21T17:10:55.230 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.224794 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-02-21T17:10:55.235 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.224939 I | op-mgr: successful modules: balancer 2022-02-21T17:10:55.256 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.251627 I | op-osd: start running osds in namespace "rook-ceph" 2022-02-21T17:10:55.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.251658 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-02-21T17:10:55.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.251670 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-02-21T17:10:55.319 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.313887 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-02-21T17:10:55.323 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.318125 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-02-21T17:10:55.323 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.318151 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-02-21T17:10:55.323 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.318156 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-02-21T17:10:55.900 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.894904 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-02-21T17:10:55.900 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.894934 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T17:10:55.912 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.907361 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-02-21T17:10:55.912 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.907394 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-02-21T17:10:55.913 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:55.907407 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-02-21T17:10:56.099 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:56.094565 I | op-mgr: successful modules: prometheus 2022-02-21T17:10:56.102 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:56.097539 I | op-mgr: successful modules: dashboard 2022-02-21T17:10:56.122 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:10:56.117277 I | op-mgr: successful modules: mgr module(s) from the spec 2022-02-21T17:11:02.361 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status -f json 2022-02-21T17:11:04.309 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:04.304208 I | op-mgr: successful modules: orchestrator modules 2022-02-21T17:11:04.376 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:11:04.377 INFO:teuthology.orchestra.run.smithi005.stdout:{"available": true, "backend": "rook"} 2022-02-21T17:11:04.404 INFO:tasks.rook: mgr/rook orchestrator is active 2022-02-21T17:11:04.405 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph config set mgr mgr/rook/storage_class scratch 2022-02-21T17:11:05.165 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch apply osd --all-available-devices 2022-02-21T17:11:05.915 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.908667 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-02-21T17:11:05.915 INFO:tasks.rook.operator.smithi005.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", AllowUnsupported: true}, 2022-02-21T17:11:05.915 INFO:tasks.rook.operator.smithi005.stdout: Storage: v1.StorageScopeSpec{ 2022-02-21T17:11:05.916 INFO:tasks.rook.operator.smithi005.stdout: ... // 3 identical fields 2022-02-21T17:11:05.916 INFO:tasks.rook.operator.smithi005.stdout: Config: nil, 2022-02-21T17:11:05.916 INFO:tasks.rook.operator.smithi005.stdout: Selection: {}, 2022-02-21T17:11:05.916 INFO:tasks.rook.operator.smithi005.stdout:- StorageClassDeviceSets: nil, 2022-02-21T17:11:05.916 INFO:tasks.rook.operator.smithi005.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-02-21T17:11:05.917 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.917 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi005-lv-1", 2022-02-21T17:11:05.917 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.917 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.918 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.918 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.918 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi005-lv-2", 2022-02-21T17:11:05.918 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.918 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.919 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.919 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.919 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi005-lv-3", 2022-02-21T17:11:05.920 INFO:teuthology.orchestra.run.smithi005.stdout:['Success'] 2022-02-21T17:11:05.925 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.925 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.925 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.926 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.926 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi005-lv-4", 2022-02-21T17:11:05.926 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.926 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.927 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.927 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.927 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi033-lv-1", 2022-02-21T17:11:05.927 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.927 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.928 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.928 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.928 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi033-lv-2", 2022-02-21T17:11:05.928 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.929 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.929 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.929 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.929 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi033-lv-3", 2022-02-21T17:11:05.929 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.930 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.930 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.930 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.930 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi033-lv-4", 2022-02-21T17:11:05.930 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.931 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.931 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.931 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.931 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi067-lv-1", 2022-02-21T17:11:05.931 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.932 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.932 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.932 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.932 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi067-lv-2", 2022-02-21T17:11:05.933 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.933 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.933 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.933 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.933 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi067-lv-3", 2022-02-21T17:11:05.934 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.934 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.934 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.934 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:11:05.935 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi067-lv-4", 2022-02-21T17:11:05.935 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:11:05.935 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-02-21T17:11:05.935 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.935 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:11:05.936 INFO:tasks.rook.operator.smithi005.stdout: }, 2022-02-21T17:11:05.936 INFO:tasks.rook.operator.smithi005.stdout: Annotations: nil, 2022-02-21T17:11:05.936 INFO:tasks.rook.operator.smithi005.stdout: Labels: nil, 2022-02-21T17:11:05.936 INFO:tasks.rook.operator.smithi005.stdout: ... // 20 identical fields 2022-02-21T17:11:05.936 INFO:tasks.rook.operator.smithi005.stdout: } 2022-02-21T17:11:05.937 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.908847 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-02-21T17:11:05.937 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.909217 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-02-21T17:11:05.937 INFO:tasks.rook.operator.smithi005.stdout:I0221 17:11:05.909268 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-02-21T17:11:05.937 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.909264 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-02-21T17:11:05.938 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.909306 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-02-21T17:11:05.938 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.909341 I | operator: successfully started the controller-runtime manager 2022-02-21T17:11:05.939 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.934003 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T17:11:05.939 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.934043 I | operator: watching all namespaces for Ceph CRs 2022-02-21T17:11:05.939 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.934095 I | operator: setting up schemes 2022-02-21T17:11:05.940 INFO:tasks.rook:Waiting for 12 OSDs 2022-02-21T17:11:05.941 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:11:05.945 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:05.940865 I | operator: setting up the controller-runtime manager 2022-02-21T17:11:06.150 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.145443 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T17:11:06.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148526 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T17:11:06.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148561 I | ceph-cluster-controller: successfully started 2022-02-21T17:11:06.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148618 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T17:11:06.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148643 I | ceph-crashcollector-controller: successfully started 2022-02-21T17:11:06.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148672 I | ceph-block-pool-controller: successfully started 2022-02-21T17:11:06.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148704 I | ceph-object-store-user-controller: successfully started 2022-02-21T17:11:06.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148733 I | ceph-object-realm-controller: successfully started 2022-02-21T17:11:06.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148753 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T17:11:06.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148772 I | ceph-object-zone-controller: successfully started 2022-02-21T17:11:06.158 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148910 I | ceph-object-controller: successfully started 2022-02-21T17:11:06.158 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148955 I | ceph-file-controller: successfully started 2022-02-21T17:11:06.158 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.148994 I | ceph-nfs-controller: successfully started 2022-02-21T17:11:06.158 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149031 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T17:11:06.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149062 I | ceph-client-controller: successfully started 2022-02-21T17:11:06.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149090 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T17:11:06.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149133 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T17:11:06.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149154 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T17:11:06.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149167 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T17:11:06.160 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149179 I | ceph-bucket-topic: successfully started 2022-02-21T17:11:06.160 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149192 I | ceph-bucket-notification: successfully started 2022-02-21T17:11:06.160 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149202 I | ceph-bucket-notification: successfully started 2022-02-21T17:11:06.160 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.149210 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T17:11:06.161 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.150419 I | operator: starting the controller-runtime manager 2022-02-21T17:11:06.258 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.253644 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T17:11:06.263 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.255676 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T17:11:06.263 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.255745 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T17:11:06.264 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.255767 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T17:11:06.265 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.258354 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T17:11:06.265 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.259563 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:11:06.267 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.260801 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:11:06.268 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.260858 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-02-21T17:11:06.268 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.260877 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-02-21T17:11:06.268 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.262838 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-02-21T17:11:06.268 INFO:tasks.rook.operator.smithi005.stdout:I0221 17:11:06.262916 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-02-21T17:11:06.284 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.279648 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-02-21T17:11:06.287 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.281924 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T17:11:06.307 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300103 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-02-21T17:11:06.307 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300126 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-02-21T17:11:06.307 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300131 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-02-21T17:11:06.308 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300136 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-02-21T17:11:06.308 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300141 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-02-21T17:11:06.308 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300147 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-02-21T17:11:06.309 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300153 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-02-21T17:11:06.309 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300159 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-02-21T17:11:06.309 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300164 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-02-21T17:11:06.309 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300170 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-02-21T17:11:06.310 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300175 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-02-21T17:11:06.310 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300180 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-02-21T17:11:06.310 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300186 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-02-21T17:11:06.310 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300191 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-02-21T17:11:06.310 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300196 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-02-21T17:11:06.311 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300202 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-02-21T17:11:06.311 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300250 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:11:06.311 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:06.300269 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:11:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:11:06.509 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-02-21T17:11:06.510 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T17:11:11.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490388 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-02-21T17:11:11.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490433 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-02-21T17:11:11.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490448 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T17:11:11.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490455 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T17:11:11.497 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490463 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T17:11:11.497 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490469 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T17:11:11.499 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490475 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T17:11:11.499 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490481 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T17:11:11.499 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490489 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T17:11:11.499 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490495 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T17:11:11.500 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490505 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T17:11:11.500 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490518 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T17:11:11.500 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490528 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-02-21T17:11:11.500 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490539 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-02-21T17:11:11.500 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490550 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-02-21T17:11:11.501 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490557 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-02-21T17:11:11.501 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490564 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-02-21T17:11:11.501 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490570 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-02-21T17:11:11.501 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490577 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-02-21T17:11:11.502 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490588 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T17:11:11.502 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490604 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T17:11:11.502 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490616 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-02-21T17:11:11.502 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490625 I | ceph-csi: Kubernetes version is 1.23 2022-02-21T17:11:11.503 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490633 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-02-21T17:11:11.503 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.490641 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-02-21T17:11:11.529 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.523581 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-02-21T17:11:11.538 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.532085 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:11:11.538 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.532111 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:11:11.538 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.532117 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:11:11.538 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.532122 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:11:11.539 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.532128 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:11:11.539 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.532133 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:11:11.539 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.532138 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-02-21T17:11:11.691 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.684876 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:11:11.691 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.684902 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:11:11.692 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.684908 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-02-21T17:11:11.737 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.732848 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-02-21T17:11:11.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.943868 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:11:11.949 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.943897 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:11:11.949 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:11.943906 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-02-21T17:11:12.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.582735 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T17:11:12.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.582764 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T17:11:12.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.582779 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T17:11:12.657 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.651704 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:11:12.658 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.651731 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:11:12.658 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.651739 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-02-21T17:11:12.756 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.751758 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:11:12.790 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.785494 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:11:12.791 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.785711 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:11:12.798 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:12.792872 I | ceph-csi: successfully started CSI CephFS driver 2022-02-21T17:11:13.108 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:13.103992 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-02-21T17:11:13.164 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:13.159673 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-02-21T17:11:13.164 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:13.159715 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-02-21T17:11:13.170 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:13.166312 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-02-21T17:11:13.210 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:13.204894 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T17:11:13.328 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:13.323012 I | op-mon: start running mons 2022-02-21T17:11:13.395 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:13.390275 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:11:13.571 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:13.566266 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789","10.106.73.178:6789","10.109.91.196:6789"]}] data:a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:2] 2022-02-21T17:11:14.167 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:14.162504 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:11:14.167 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:14.162800 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:11:15.770 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:15.765382 I | op-mon: targeting the mon count 3 2022-02-21T17:11:15.779 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:15.770623 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:11:16.153 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:16.148530 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:11:16.154 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:16.148566 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:11:16.509 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:16.503991 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:11:16.509 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:16.504032 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:11:16.511 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:11:16.861 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:16.856137 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:11:16.862 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:16.856173 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:11:17.006 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:11:17.026 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-02-21T17:11:17.027 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T17:11:17.226 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:17.221212 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:11:17.226 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:17.221247 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T17:11:17.569 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:17.564827 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T17:11:17.671 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:17.564858 I | op-mon: checking for basic quorum with existing mons 2022-02-21T17:11:17.686 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:17.682133 I | op-mon: mon "a" endpoint is [v2:10.110.173.53:3300,v1:10.110.173.53:6789] 2022-02-21T17:11:17.805 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:17.800390 I | op-mon: mon "b" endpoint is [v2:10.106.73.178:3300,v1:10.106.73.178:6789] 2022-02-21T17:11:18.169 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:18.164692 I | op-mon: mon "c" endpoint is [v2:10.109.91.196:3300,v1:10.109.91.196:6789] 2022-02-21T17:11:18.769 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:18.764158 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789","10.106.73.178:6789","10.109.91.196:6789"]}] data:a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:2] 2022-02-21T17:11:19.368 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:19.364157 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:11:19.371 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:19.364423 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:11:19.788 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:19.783417 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T17:11:19.798 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:19.793414 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T17:11:19.798 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:19.793439 I | op-mon: waiting for mon quorum with [a b c] 2022-02-21T17:11:20.373 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:20.368777 I | op-mon: mons running: [a b c] 2022-02-21T17:11:20.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:20.793111 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:11:20.817 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:20.813254 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-02-21T17:11:20.827 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:20.822819 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-02-21T17:11:20.828 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:20.822848 I | op-mon: waiting for mon quorum with [a b c] 2022-02-21T17:11:20.972 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:20.967713 I | op-mon: mons running: [a b c] 2022-02-21T17:11:21.389 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:21.384609 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:11:21.396 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:21.390313 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-02-21T17:11:21.402 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:21.397621 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-02-21T17:11:21.402 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:21.397640 I | op-mon: waiting for mon quorum with [a b c] 2022-02-21T17:11:21.573 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:21.568132 I | op-mon: mons running: [a b c] 2022-02-21T17:11:22.009 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:22.004187 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:11:22.010 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:22.004218 I | op-mon: mons created: 3 2022-02-21T17:11:22.431 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:22.426521 I | op-mon: waiting for mon quorum with [a b c] 2022-02-21T17:11:22.455 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:22.450881 I | op-mon: mons running: [a b c] 2022-02-21T17:11:22.872 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:22.867079 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:11:22.872 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:22.867126 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-02-21T17:11:23.319 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:23.315094 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-02-21T17:11:23.748 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:23.743359 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-02-21T17:11:24.173 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:24.169159 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-02-21T17:11:24.635 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:24.630724 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-02-21T17:11:24.636 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:24.630754 I | cephclient: getting or creating ceph auth key "client.crash" 2022-02-21T17:11:25.085 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:25.080951 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-02-21T17:11:25.493 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:25.488837 I | cephclient: successfully enabled msgr2 protocol 2022-02-21T17:11:25.494 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:25.488877 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-02-21T17:11:25.494 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:25.488891 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-02-21T17:11:25.913 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:25.908967 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-02-21T17:11:25.947 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:25.943441 I | op-mgr: start running mgr 2022-02-21T17:11:25.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:25.943474 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-02-21T17:11:26.420 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:26.416101 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-02-21T17:11:26.431 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:26.426936 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-02-21T17:11:26.434 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:26.430253 I | op-mgr: setting services to point to mgr "a" 2022-02-21T17:11:26.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:26.602720 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-02-21T17:11:26.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:26.602851 I | op-mgr: successful modules: balancer 2022-02-21T17:11:26.617 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:26.612643 I | op-osd: start running osds in namespace "rook-ceph" 2022-02-21T17:11:26.617 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:26.612685 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-02-21T17:11:26.707 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:26.702201 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-02-21T17:11:26.829 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:26.824293 I | op-osd: creating 1 new PVCs for device set "smithi005-lv-1" 2022-02-21T17:11:27.011 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.006364 I | op-osd: successfully provisioned PVC "smithi005-lv-1-data-0mcn2z" 2022-02-21T17:11:27.011 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.006384 I | op-osd: creating 1 new PVCs for device set "smithi005-lv-2" 2022-02-21T17:11:27.029 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:11:27.230 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.225978 I | op-osd: successfully provisioned PVC "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:11:27.231 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.226018 I | op-osd: creating 1 new PVCs for device set "smithi005-lv-3" 2022-02-21T17:11:27.408 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.403560 I | op-osd: successfully provisioned PVC "smithi005-lv-3-data-0glx9j" 2022-02-21T17:11:27.409 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.403614 I | op-osd: creating 1 new PVCs for device set "smithi005-lv-4" 2022-02-21T17:11:27.559 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:11:27.578 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-02-21T17:11:27.579 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T17:11:27.655 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.650898 I | op-osd: successfully provisioned PVC "smithi005-lv-4-data-09b8xq" 2022-02-21T17:11:27.655 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.650933 I | op-osd: creating 1 new PVCs for device set "smithi033-lv-1" 2022-02-21T17:11:27.823 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.818811 I | op-osd: successfully provisioned PVC "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:11:27.824 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.818840 I | op-osd: creating 1 new PVCs for device set "smithi033-lv-2" 2022-02-21T17:11:27.971 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.967025 I | op-mgr: successful modules: dashboard 2022-02-21T17:11:27.976 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.967887 I | op-mgr: successful modules: mgr module(s) from the spec 2022-02-21T17:11:27.977 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:27.972098 I | op-mgr: successful modules: prometheus 2022-02-21T17:11:28.008 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.003678 I | op-osd: successfully provisioned PVC "smithi033-lv-2-data-0drpqd" 2022-02-21T17:11:28.008 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.003725 I | op-osd: creating 1 new PVCs for device set "smithi033-lv-3" 2022-02-21T17:11:28.208 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.203408 I | op-osd: successfully provisioned PVC "smithi033-lv-3-data-0t898k" 2022-02-21T17:11:28.208 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.203456 I | op-osd: creating 1 new PVCs for device set "smithi033-lv-4" 2022-02-21T17:11:28.409 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.404660 I | op-osd: successfully provisioned PVC "smithi033-lv-4-data-0p96d6" 2022-02-21T17:11:28.409 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.404694 I | op-osd: creating 1 new PVCs for device set "smithi067-lv-1" 2022-02-21T17:11:28.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.602811 I | op-osd: successfully provisioned PVC "smithi067-lv-1-data-05cnzw" 2022-02-21T17:11:28.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.602857 I | op-osd: creating 1 new PVCs for device set "smithi067-lv-2" 2022-02-21T17:11:28.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.820479 I | op-osd: successfully provisioned PVC "smithi067-lv-2-data-08phkg" 2022-02-21T17:11:28.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:28.820517 I | op-osd: creating 1 new PVCs for device set "smithi067-lv-3" 2022-02-21T17:11:29.007 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:29.002734 I | op-osd: successfully provisioned PVC "smithi067-lv-3-data-0d6rns" 2022-02-21T17:11:29.007 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:29.002771 I | op-osd: creating 1 new PVCs for device set "smithi067-lv-4" 2022-02-21T17:11:29.212 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:29.207775 I | op-osd: successfully provisioned PVC "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:11:29.226 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:29.222100 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-1-data-0mcn2z" 2022-02-21T17:11:29.614 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:29.609641 I | op-mgr: successful modules: orchestrator modules 2022-02-21T17:11:29.622 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:29.617852 I | op-osd: started OSD provisioning job for PVC "smithi005-lv-1-data-0mcn2z" 2022-02-21T17:11:29.622 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:29.617884 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:11:30.058 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:30.053844 I | op-osd: started OSD provisioning job for PVC "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:11:30.058 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:30.053884 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-3-data-0glx9j" 2022-02-21T17:11:30.810 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:30.805180 I | op-osd: started OSD provisioning job for PVC "smithi005-lv-3-data-0glx9j" 2022-02-21T17:11:30.810 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:30.805227 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-4-data-09b8xq" 2022-02-21T17:11:31.172 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:31.167698 I | op-osd: started OSD provisioning job for PVC "smithi005-lv-4-data-09b8xq" 2022-02-21T17:11:31.172 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:31.167734 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:11:31.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:31.821177 I | op-osd: started OSD provisioning job for PVC "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:11:31.826 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:31.821213 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-2-data-0drpqd" 2022-02-21T17:11:32.230 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:32.225296 I | op-osd: started OSD provisioning job for PVC "smithi033-lv-2-data-0drpqd" 2022-02-21T17:11:32.230 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:32.225333 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-3-data-0t898k" 2022-02-21T17:11:33.070 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:33.065707 I | op-osd: started OSD provisioning job for PVC "smithi033-lv-3-data-0t898k" 2022-02-21T17:11:33.070 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:33.065746 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-4-data-0p96d6" 2022-02-21T17:11:33.740 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:33.735374 I | op-osd: started OSD provisioning job for PVC "smithi033-lv-4-data-0p96d6" 2022-02-21T17:11:33.740 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:33.735411 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-1-data-05cnzw" 2022-02-21T17:11:34.447 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:34.443304 I | op-osd: started OSD provisioning job for PVC "smithi067-lv-1-data-05cnzw" 2022-02-21T17:11:34.448 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:34.443372 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-2-data-08phkg" 2022-02-21T17:11:34.799 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:34.794917 I | op-osd: started OSD provisioning job for PVC "smithi067-lv-2-data-08phkg" 2022-02-21T17:11:34.800 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:34.794977 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-3-data-0d6rns" 2022-02-21T17:11:35.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:35.597430 I | op-osd: started OSD provisioning job for PVC "smithi067-lv-3-data-0d6rns" 2022-02-21T17:11:35.602 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:35.597477 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:11:36.114 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.109675 I | op-osd: started OSD provisioning job for PVC "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:11:36.114 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.109725 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-02-21T17:11:36.115 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.109738 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-02-21T17:11:36.946 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940751 I | op-osd: OSD orchestration status for PVC smithi005-lv-1-data-0mcn2z is "starting" 2022-02-21T17:11:36.946 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940789 I | op-osd: OSD orchestration status for PVC smithi005-lv-2-data-0hnmnc is "starting" 2022-02-21T17:11:36.947 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940800 I | op-osd: OSD orchestration status for PVC smithi005-lv-3-data-0glx9j is "starting" 2022-02-21T17:11:36.947 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940808 I | op-osd: OSD orchestration status for PVC smithi005-lv-4-data-09b8xq is "starting" 2022-02-21T17:11:36.947 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940816 I | op-osd: OSD orchestration status for PVC smithi033-lv-1-data-0bqbtn is "starting" 2022-02-21T17:11:36.947 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940823 I | op-osd: OSD orchestration status for PVC smithi033-lv-2-data-0drpqd is "starting" 2022-02-21T17:11:36.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940831 I | op-osd: OSD orchestration status for PVC smithi033-lv-3-data-0t898k is "starting" 2022-02-21T17:11:36.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940840 I | op-osd: OSD orchestration status for PVC smithi033-lv-4-data-0p96d6 is "starting" 2022-02-21T17:11:36.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940849 I | op-osd: OSD orchestration status for PVC smithi067-lv-1-data-05cnzw is "starting" 2022-02-21T17:11:36.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940863 I | op-osd: OSD orchestration status for PVC smithi067-lv-2-data-08phkg is "starting" 2022-02-21T17:11:36.949 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940872 I | op-osd: OSD orchestration status for PVC smithi067-lv-3-data-0d6rns is "starting" 2022-02-21T17:11:36.949 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:36.940880 I | op-osd: OSD orchestration status for PVC smithi067-lv-4-data-0kh9ql is "starting" 2022-02-21T17:11:37.580 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:11:38.096 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:11:38.115 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-02-21T17:11:38.115 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T17:11:44.557 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:44.552536 I | op-osd: OSD orchestration status for node smithi067-lv-1-data-05cnzw is "orchestrating" 2022-02-21T17:11:44.557 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:44.552569 I | op-osd: OSD orchestration status for node smithi005-lv-1-data-0mcn2z is "orchestrating" 2022-02-21T17:11:45.545 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:45.539931 I | op-osd: OSD orchestration status for PVC smithi005-lv-1-data-0mcn2z is "orchestrating" 2022-02-21T17:11:45.552 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:45.547984 I | op-osd: OSD orchestration status for PVC smithi067-lv-1-data-05cnzw is "orchestrating" 2022-02-21T17:11:45.630 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:45.625955 I | op-osd: OSD orchestration status for node smithi067-lv-3-data-0d6rns is "orchestrating" 2022-02-21T17:11:46.657 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:46.653108 I | op-osd: OSD orchestration status for PVC smithi067-lv-3-data-0d6rns is "orchestrating" 2022-02-21T17:11:48.010 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:48.006674 I | op-osd: OSD orchestration status for node smithi005-lv-2-data-0hnmnc is "orchestrating" 2022-02-21T17:11:48.116 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:11:48.313 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:48.309466 I | op-osd: OSD orchestration status for node smithi005-lv-3-data-0glx9j is "orchestrating" 2022-02-21T17:11:48.584 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:48.580093 I | op-osd: OSD orchestration status for PVC smithi005-lv-2-data-0hnmnc is "orchestrating" 2022-02-21T17:11:48.587 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:11:48.606 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":3,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1645463507,"num_remapped_pgs":0} 2022-02-21T17:11:48.606 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T17:11:48.713 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:48.709080 I | op-osd: OSD orchestration status for PVC smithi005-lv-3-data-0glx9j is "orchestrating" 2022-02-21T17:11:53.479 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:53.474881 I | op-osd: OSD orchestration status for PVC smithi005-lv-1-data-0mcn2z is "completed" 2022-02-21T17:11:53.479 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:53.474917 I | op-osd: creating OSD 0 on PVC "smithi005-lv-1-data-0mcn2z" 2022-02-21T17:11:53.479 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:53.474931 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-1-data-0mcn2z" 2022-02-21T17:11:54.528 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:54.522202 I | op-osd: OSD orchestration status for PVC smithi067-lv-1-data-05cnzw is "completed" 2022-02-21T17:11:54.529 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:54.522242 I | op-osd: creating OSD 2 on PVC "smithi067-lv-1-data-05cnzw" 2022-02-21T17:11:54.529 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:54.522265 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-1-data-05cnzw" 2022-02-21T17:11:54.974 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:54.969610 I | op-osd: OSD orchestration status for PVC smithi067-lv-3-data-0d6rns is "completed" 2022-02-21T17:11:54.974 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:54.969662 I | op-osd: creating OSD 1 on PVC "smithi067-lv-3-data-0d6rns" 2022-02-21T17:11:54.974 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:54.969690 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-3-data-0d6rns" 2022-02-21T17:11:58.134 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:58.129966 I | op-osd: OSD orchestration status for PVC smithi005-lv-3-data-0glx9j is "completed" 2022-02-21T17:11:58.134 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:58.130007 I | op-osd: creating OSD 3 on PVC "smithi005-lv-3-data-0glx9j" 2022-02-21T17:11:58.134 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:58.130028 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-3-data-0glx9j" 2022-02-21T17:11:58.608 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:11:59.416 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:59.411878 I | op-osd: OSD orchestration status for PVC smithi005-lv-2-data-0hnmnc is "completed" 2022-02-21T17:11:59.417 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:59.411912 I | op-osd: creating OSD 4 on PVC "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:11:59.417 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:11:59.411925 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:11:59.828 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:11:59.847 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":7,"num_osds":5,"num_up_osds":0,"osd_up_since":0,"num_in_osds":5,"osd_in_since":1645463512,"num_remapped_pgs":0} 2022-02-21T17:11:59.848 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T17:12:01.392 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:01.384026 I | op-osd: OSD orchestration status for node smithi067-lv-4-data-0kh9ql is "orchestrating" 2022-02-21T17:12:01.392 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:01.384281 I | op-osd: OSD orchestration status for node smithi067-lv-2-data-08phkg is "orchestrating" 2022-02-21T17:12:01.392 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:01.384436 I | op-osd: OSD orchestration status for node smithi005-lv-4-data-09b8xq is "orchestrating" 2022-02-21T17:12:01.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:01.384627 I | op-osd: OSD orchestration status for node smithi033-lv-3-data-0t898k is "orchestrating" 2022-02-21T17:12:01.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:01.384645 I | op-osd: OSD orchestration status for PVC smithi067-lv-2-data-08phkg is "orchestrating" 2022-02-21T17:12:01.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:01.384840 I | op-osd: OSD orchestration status for PVC smithi067-lv-4-data-0kh9ql is "orchestrating" 2022-02-21T17:12:01.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:01.384851 I | op-osd: OSD orchestration status for node smithi033-lv-1-data-0bqbtn is "orchestrating" 2022-02-21T17:12:01.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:01.388315 I | op-osd: OSD orchestration status for PVC smithi033-lv-3-data-0t898k is "orchestrating" 2022-02-21T17:12:01.808 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:01.804612 I | op-osd: OSD orchestration status for PVC smithi005-lv-4-data-09b8xq is "orchestrating" 2022-02-21T17:12:02.518 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:02.514641 I | op-osd: OSD orchestration status for PVC smithi033-lv-1-data-0bqbtn is "orchestrating" 2022-02-21T17:12:03.097 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:03.093614 I | op-osd: OSD orchestration status for node smithi033-lv-4-data-0p96d6 is "orchestrating" 2022-02-21T17:12:03.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:03.093668 I | op-osd: OSD orchestration status for node smithi033-lv-2-data-0drpqd is "orchestrating" 2022-02-21T17:12:04.154 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:04.150379 I | op-osd: OSD orchestration status for PVC smithi033-lv-2-data-0drpqd is "orchestrating" 2022-02-21T17:12:04.155 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:04.150451 I | op-osd: OSD orchestration status for PVC smithi033-lv-4-data-0p96d6 is "orchestrating" 2022-02-21T17:12:07.535 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:07.517519 I | op-osd: OSD orchestration status for PVC smithi033-lv-4-data-0p96d6 is "failed" 2022-02-21T17:12:07.536 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:07.517613 E | op-osd: failed to provision OSD(s) on PVC smithi033-lv-4-data-0p96d6. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-02-21T17:12:07.536 INFO:tasks.rook.operator.smithi005.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7bc39c35-54dc-44eb-8410-1acb55a250b9 2022-02-21T17:12:07.536 INFO:tasks.rook.operator.smithi005.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-02-21T17:12:07.536 INFO:tasks.rook.operator.smithi005.stdout: stderr: 2022-02-21T17:12:07.132+0000 7f968d336700 -1 failed for service _ceph-mon._tcp 2022-02-21T17:12:07.537 INFO:tasks.rook.operator.smithi005.stdout: stderr: 2022-02-21T17:12:07.132+0000 7f968d336700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-02-21T17:12:07.537 INFO:tasks.rook.operator.smithi005.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-02-21T17:12:07.537 INFO:tasks.rook.operator.smithi005.stdout:Traceback (most recent call last): 2022-02-21T17:12:07.537 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-02-21T17:12:07.537 INFO:tasks.rook.operator.smithi005.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-02-21T17:12:07.538 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-02-21T17:12:07.538 INFO:tasks.rook.operator.smithi005.stdout: self.main(self.argv) 2022-02-21T17:12:07.538 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-02-21T17:12:07.538 INFO:tasks.rook.operator.smithi005.stdout: return f(*a, **kw) 2022-02-21T17:12:07.539 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-02-21T17:12:07.539 INFO:tasks.rook.operator.smithi005.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-02-21T17:12:07.539 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-02-21T17:12:07.539 INFO:tasks.rook.operator.smithi005.stdout: instance.main() 2022-02-21T17:12:07.539 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-02-21T17:12:07.540 INFO:tasks.rook.operator.smithi005.stdout: terminal.dispatch(self.mapper, self.argv) 2022-02-21T17:12:07.540 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-02-21T17:12:07.540 INFO:tasks.rook.operator.smithi005.stdout: instance.main() 2022-02-21T17:12:07.540 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-02-21T17:12:07.541 INFO:tasks.rook.operator.smithi005.stdout: self.safe_prepare(self.args) 2022-02-21T17:12:07.541 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-02-21T17:12:07.541 INFO:tasks.rook.operator.smithi005.stdout: self.prepare() 2022-02-21T17:12:07.541 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-02-21T17:12:07.541 INFO:tasks.rook.operator.smithi005.stdout: return func(*a, **kw) 2022-02-21T17:12:07.542 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-02-21T17:12:07.542 INFO:tasks.rook.operator.smithi005.stdout: osd_fsid, json.dumps(secrets)) 2022-02-21T17:12:07.542 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-02-21T17:12:07.542 INFO:tasks.rook.operator.smithi005.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-02-21T17:12:07.543 INFO:tasks.rook.operator.smithi005.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-02-21T17:12:07.543 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,642][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi033-lv-4-data-0p96d6 raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_4 2022-02-21T17:12:07.543 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,643][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-02-21T17:12:07.543 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-02-21T17:12:07.543 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-02-21T17:12:07.544 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-02-21T17:12:07.544 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-02-21T17:12:07.544 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-02-21T17:12:07.544 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-02-21T17:12:07.544 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-02-21T17:12:07.545 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-02-21T17:12:07.545 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-02-21T17:12:07.545 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-02-21T17:12:07.545 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-02-21T17:12:07.546 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-02-21T17:12:07.546 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-02-21T17:12:07.546 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-02-21T17:12:07.546 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-02-21T17:12:07.546 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,651][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-02-21T17:12:07.547 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,660][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-02-21T17:12:07.547 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,661][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_4 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-02-21T17:12:07.547 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,967][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_4 2022-02-21T17:12:07.547 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,972][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_4" KNAME="dm-3" MAJ:MIN="253:3" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-02-21T17:12:07.548 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,972][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_4 2022-02-21T17:12:07.548 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,976][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_4 to check for BlueStore label 2022-02-21T17:12:07.548 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,977][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_4 to check for BlueStore label 2022-02-21T17:12:07.548 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,977][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_4 2022-02-21T17:12:07.548 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/vg_nvme/lv_4 /dev/mapper/vg_nvme-lv_4 /dev/disk/by-id/dm-uuid-LVM-jsx2ApcoCeWWgQ6qig11ucIXmmNcKAgxDIn32xK7UMRLa7mnWznX3SqOIPOM2wLG /dev/disk/by-id/dm-name-vg_nvme-lv_4 2022-02-21T17:12:07.549 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-3 2022-02-21T17:12:07.549 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-3 2022-02-21T17:12:07.549 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-02-21T17:12:07.549 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_4 2022-02-21T17:12:07.550 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_4 2022-02-21T17:12:07.550 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-02-21T17:12:07.550 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-02-21T17:12:07.550 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-02-21T17:12:07.551 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-02-21T17:12:07.551 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-02-21T17:12:07.551 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UUID=LVM-jsx2ApcoCeWWgQ6qig11ucIXmmNcKAgxDIn32xK7UMRLa7mnWznX3SqOIPOM2wLG 2022-02-21T17:12:07.551 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-02-21T17:12:07.551 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout MAJOR=253 2022-02-21T17:12:07.552 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout MINOR=3 2022-02-21T17:12:07.552 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-02-21T17:12:07.552 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-02-21T17:12:07.552 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=211151947 2022-02-21T17:12:07.552 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-02-21T17:12:07.553 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,003][ceph_volume.process][INFO ] stdout AQDnxxNi10sjABAA5AeZEgrF7DTkf6J29dYXsg== 2022-02-21T17:12:07.553 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,004][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7bc39c35-54dc-44eb-8410-1acb55a250b9 2022-02-21T17:12:07.553 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,138][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-02-21T17:12:07.553 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,138][ceph_volume.process][INFO ] stderr 2022-02-21T17:12:07.132+0000 7f968d336700 -1 failed for service _ceph-mon._tcp 2022-02-21T17:12:07.554 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,139][ceph_volume.process][INFO ] stderr 2022-02-21T17:12:07.132+0000 7f968d336700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-02-21T17:12:07.554 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,139][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-02-21T17:12:07.554 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,139][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-02-21T17:12:07.554 INFO:tasks.rook.operator.smithi005.stdout:Traceback (most recent call last): 2022-02-21T17:12:07.555 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-02-21T17:12:07.555 INFO:tasks.rook.operator.smithi005.stdout: self.prepare() 2022-02-21T17:12:07.555 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-02-21T17:12:07.555 INFO:tasks.rook.operator.smithi005.stdout: return func(*a, **kw) 2022-02-21T17:12:07.556 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-02-21T17:12:07.556 INFO:tasks.rook.operator.smithi005.stdout: osd_fsid, json.dumps(secrets)) 2022-02-21T17:12:07.556 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-02-21T17:12:07.556 INFO:tasks.rook.operator.smithi005.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-02-21T17:12:07.556 INFO:tasks.rook.operator.smithi005.stdout:RuntimeError: Unable to create a new OSD id 2022-02-21T17:12:07.557 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,140][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-02-21T17:12:07.557 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,140][ceph_volume][ERROR ] exception caught by decorator 2022-02-21T17:12:07.557 INFO:tasks.rook.operator.smithi005.stdout:Traceback (most recent call last): 2022-02-21T17:12:07.557 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-02-21T17:12:07.557 INFO:tasks.rook.operator.smithi005.stdout: return f(*a, **kw) 2022-02-21T17:12:07.558 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-02-21T17:12:07.558 INFO:tasks.rook.operator.smithi005.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-02-21T17:12:07.558 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-02-21T17:12:07.558 INFO:tasks.rook.operator.smithi005.stdout: instance.main() 2022-02-21T17:12:07.559 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-02-21T17:12:07.559 INFO:tasks.rook.operator.smithi005.stdout: terminal.dispatch(self.mapper, self.argv) 2022-02-21T17:12:07.559 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-02-21T17:12:07.559 INFO:tasks.rook.operator.smithi005.stdout: instance.main() 2022-02-21T17:12:07.559 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-02-21T17:12:07.560 INFO:tasks.rook.operator.smithi005.stdout: self.safe_prepare(self.args) 2022-02-21T17:12:07.560 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-02-21T17:12:07.560 INFO:tasks.rook.operator.smithi005.stdout: self.prepare() 2022-02-21T17:12:07.560 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-02-21T17:12:07.561 INFO:tasks.rook.operator.smithi005.stdout: return func(*a, **kw) 2022-02-21T17:12:07.561 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-02-21T17:12:07.561 INFO:tasks.rook.operator.smithi005.stdout: osd_fsid, json.dumps(secrets)) 2022-02-21T17:12:07.561 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-02-21T17:12:07.561 INFO:tasks.rook.operator.smithi005.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-02-21T17:12:07.562 INFO:tasks.rook.operator.smithi005.stdout:RuntimeError: Unable to create a new OSD id 2022-02-21T17:12:07.562 INFO:tasks.rook.operator.smithi005.stdout:: exit status 1} 2022-02-21T17:12:08.547 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:08.543306 I | op-osd: OSD orchestration status for PVC smithi067-lv-2-data-08phkg is "completed" 2022-02-21T17:12:08.548 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:08.543388 I | op-osd: creating OSD 5 on PVC "smithi067-lv-2-data-08phkg" 2022-02-21T17:12:08.548 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:08.543409 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-2-data-08phkg" 2022-02-21T17:12:09.663 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:09.607608 I | op-osd: OSD orchestration status for PVC smithi033-lv-3-data-0t898k is "completed" 2022-02-21T17:12:09.664 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:09.607640 I | op-osd: creating OSD 6 on PVC "smithi033-lv-3-data-0t898k" 2022-02-21T17:12:09.664 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:09.607655 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-3-data-0t898k" 2022-02-21T17:12:09.849 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:12:11.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:11.049942 I | op-osd: OSD orchestration status for PVC smithi033-lv-1-data-0bqbtn is "completed" 2022-02-21T17:12:11.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:11.049972 I | op-osd: creating OSD 7 on PVC "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:12:11.055 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:11.049985 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:12:11.317 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:12:11.347 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":13,"num_osds":11,"num_up_osds":0,"osd_up_since":0,"num_in_osds":11,"osd_in_since":1645463526,"num_remapped_pgs":0} 2022-02-21T17:12:11.348 INFO:tasks.rook: have 0/12 OSDs 2022-02-21T17:12:12.589 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:12.585559 I | op-osd: OSD orchestration status for node smithi033-lv-4-data-0p96d6 is "orchestrating" 2022-02-21T17:12:12.590 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:12.585745 I | op-osd: OSD orchestration status for PVC smithi067-lv-4-data-0kh9ql is "completed" 2022-02-21T17:12:12.590 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:12.585764 I | op-osd: creating OSD 8 on PVC "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:12:12.590 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:12.585786 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:12:14.266 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:14.262087 I | op-osd: OSD orchestration status for PVC smithi033-lv-2-data-0drpqd is "completed" 2022-02-21T17:12:14.269 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:14.262132 I | op-osd: creating OSD 9 on PVC "smithi033-lv-2-data-0drpqd" 2022-02-21T17:12:14.269 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:14.262154 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-2-data-0drpqd" 2022-02-21T17:12:16.102 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:16.097996 I | op-osd: OSD orchestration status for PVC smithi033-lv-4-data-0p96d6 is "orchestrating" 2022-02-21T17:12:16.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:16.098467 I | op-osd: OSD orchestration status for PVC smithi005-lv-4-data-09b8xq is "completed" 2022-02-21T17:12:16.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:16.098486 I | op-osd: creating OSD 10 on PVC "smithi005-lv-4-data-09b8xq" 2022-02-21T17:12:16.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:16.098498 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-4-data-09b8xq" 2022-02-21T17:12:17.902 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:17.896939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-02-21T17:12:17.902 INFO:tasks.rook.operator.smithi005.stdout:failed to provision OSD(s) on PVC smithi033-lv-4-data-0p96d6. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-02-21T17:12:17.903 INFO:tasks.rook.operator.smithi005.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7bc39c35-54dc-44eb-8410-1acb55a250b9 2022-02-21T17:12:17.903 INFO:tasks.rook.operator.smithi005.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-02-21T17:12:17.903 INFO:tasks.rook.operator.smithi005.stdout: stderr: 2022-02-21T17:12:07.132+0000 7f968d336700 -1 failed for service _ceph-mon._tcp 2022-02-21T17:12:17.903 INFO:tasks.rook.operator.smithi005.stdout: stderr: 2022-02-21T17:12:07.132+0000 7f968d336700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-02-21T17:12:17.904 INFO:tasks.rook.operator.smithi005.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-02-21T17:12:17.904 INFO:tasks.rook.operator.smithi005.stdout:Traceback (most recent call last): 2022-02-21T17:12:17.904 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-02-21T17:12:17.904 INFO:tasks.rook.operator.smithi005.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-02-21T17:12:17.904 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-02-21T17:12:17.905 INFO:tasks.rook.operator.smithi005.stdout: self.main(self.argv) 2022-02-21T17:12:17.905 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-02-21T17:12:17.905 INFO:tasks.rook.operator.smithi005.stdout: return f(*a, **kw) 2022-02-21T17:12:17.905 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-02-21T17:12:17.906 INFO:tasks.rook.operator.smithi005.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-02-21T17:12:17.906 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-02-21T17:12:17.906 INFO:tasks.rook.operator.smithi005.stdout: instance.main() 2022-02-21T17:12:17.906 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-02-21T17:12:17.907 INFO:tasks.rook.operator.smithi005.stdout: terminal.dispatch(self.mapper, self.argv) 2022-02-21T17:12:17.907 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-02-21T17:12:17.907 INFO:tasks.rook.operator.smithi005.stdout: instance.main() 2022-02-21T17:12:17.907 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-02-21T17:12:17.907 INFO:tasks.rook.operator.smithi005.stdout: self.safe_prepare(self.args) 2022-02-21T17:12:17.908 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-02-21T17:12:17.908 INFO:tasks.rook.operator.smithi005.stdout: self.prepare() 2022-02-21T17:12:17.908 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-02-21T17:12:17.908 INFO:tasks.rook.operator.smithi005.stdout: return func(*a, **kw) 2022-02-21T17:12:17.908 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-02-21T17:12:17.909 INFO:tasks.rook.operator.smithi005.stdout: osd_fsid, json.dumps(secrets)) 2022-02-21T17:12:17.909 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-02-21T17:12:17.909 INFO:tasks.rook.operator.smithi005.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-02-21T17:12:17.909 INFO:tasks.rook.operator.smithi005.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-02-21T17:12:17.910 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,642][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi033-lv-4-data-0p96d6 raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_4 2022-02-21T17:12:17.910 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,643][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-02-21T17:12:17.910 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-02-21T17:12:17.910 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-02-21T17:12:17.910 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-02-21T17:12:17.911 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-02-21T17:12:17.911 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-02-21T17:12:17.911 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-02-21T17:12:17.911 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-02-21T17:12:17.912 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-02-21T17:12:17.912 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-02-21T17:12:17.912 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-02-21T17:12:17.912 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-02-21T17:12:17.912 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-02-21T17:12:17.913 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-02-21T17:12:17.913 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-02-21T17:12:17.913 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,650][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-02-21T17:12:17.913 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,651][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-02-21T17:12:17.913 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,660][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-02-21T17:12:17.914 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,661][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_4 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-02-21T17:12:17.914 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,967][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_4 2022-02-21T17:12:17.914 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,972][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_4" KNAME="dm-3" MAJ:MIN="253:3" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-02-21T17:12:17.914 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,972][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_4 2022-02-21T17:12:17.917 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,976][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_4 to check for BlueStore label 2022-02-21T17:12:17.917 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,977][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_4 to check for BlueStore label 2022-02-21T17:12:17.917 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,977][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_4 2022-02-21T17:12:17.917 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/vg_nvme/lv_4 /dev/mapper/vg_nvme-lv_4 /dev/disk/by-id/dm-uuid-LVM-jsx2ApcoCeWWgQ6qig11ucIXmmNcKAgxDIn32xK7UMRLa7mnWznX3SqOIPOM2wLG /dev/disk/by-id/dm-name-vg_nvme-lv_4 2022-02-21T17:12:17.918 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-3 2022-02-21T17:12:17.918 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-3 2022-02-21T17:12:17.918 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-02-21T17:12:17.918 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,985][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_4 2022-02-21T17:12:17.919 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_4 2022-02-21T17:12:17.919 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-02-21T17:12:17.919 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-02-21T17:12:17.919 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-02-21T17:12:17.919 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-02-21T17:12:17.920 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-02-21T17:12:17.920 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_UUID=LVM-jsx2ApcoCeWWgQ6qig11ucIXmmNcKAgxDIn32xK7UMRLa7mnWznX3SqOIPOM2wLG 2022-02-21T17:12:17.920 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-02-21T17:12:17.920 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout MAJOR=253 2022-02-21T17:12:17.921 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout MINOR=3 2022-02-21T17:12:17.921 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-02-21T17:12:17.921 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-02-21T17:12:17.921 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=211151947 2022-02-21T17:12:17.921 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:06,986][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-02-21T17:12:17.922 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,003][ceph_volume.process][INFO ] stdout AQDnxxNi10sjABAA5AeZEgrF7DTkf6J29dYXsg== 2022-02-21T17:12:17.922 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,004][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7bc39c35-54dc-44eb-8410-1acb55a250b9 2022-02-21T17:12:17.922 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,138][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-02-21T17:12:17.922 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,138][ceph_volume.process][INFO ] stderr 2022-02-21T17:12:07.132+0000 7f968d336700 -1 failed for service _ceph-mon._tcp 2022-02-21T17:12:17.923 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,139][ceph_volume.process][INFO ] stderr 2022-02-21T17:12:07.132+0000 7f968d336700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-02-21T17:12:17.923 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,139][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-02-21T17:12:17.923 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,139][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-02-21T17:12:17.923 INFO:tasks.rook.operator.smithi005.stdout:Traceback (most recent call last): 2022-02-21T17:12:17.923 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-02-21T17:12:17.924 INFO:tasks.rook.operator.smithi005.stdout: self.prepare() 2022-02-21T17:12:17.924 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-02-21T17:12:17.924 INFO:tasks.rook.operator.smithi005.stdout: return func(*a, **kw) 2022-02-21T17:12:17.924 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-02-21T17:12:17.925 INFO:tasks.rook.operator.smithi005.stdout: osd_fsid, json.dumps(secrets)) 2022-02-21T17:12:17.925 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-02-21T17:12:17.925 INFO:tasks.rook.operator.smithi005.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-02-21T17:12:17.925 INFO:tasks.rook.operator.smithi005.stdout:RuntimeError: Unable to create a new OSD id 2022-02-21T17:12:17.925 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,140][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-02-21T17:12:17.926 INFO:tasks.rook.operator.smithi005.stdout:[2022-02-21 17:12:07,140][ceph_volume][ERROR ] exception caught by decorator 2022-02-21T17:12:17.926 INFO:tasks.rook.operator.smithi005.stdout:Traceback (most recent call last): 2022-02-21T17:12:17.926 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-02-21T17:12:17.926 INFO:tasks.rook.operator.smithi005.stdout: return f(*a, **kw) 2022-02-21T17:12:17.927 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-02-21T17:12:17.927 INFO:tasks.rook.operator.smithi005.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-02-21T17:12:17.927 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-02-21T17:12:17.927 INFO:tasks.rook.operator.smithi005.stdout: instance.main() 2022-02-21T17:12:17.927 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-02-21T17:12:17.928 INFO:tasks.rook.operator.smithi005.stdout: terminal.dispatch(self.mapper, self.argv) 2022-02-21T17:12:17.928 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-02-21T17:12:17.928 INFO:tasks.rook.operator.smithi005.stdout: instance.main() 2022-02-21T17:12:17.928 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-02-21T17:12:17.929 INFO:tasks.rook.operator.smithi005.stdout: self.safe_prepare(self.args) 2022-02-21T17:12:17.929 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-02-21T17:12:17.929 INFO:tasks.rook.operator.smithi005.stdout: self.prepare() 2022-02-21T17:12:17.929 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-02-21T17:12:17.929 INFO:tasks.rook.operator.smithi005.stdout: return func(*a, **kw) 2022-02-21T17:12:17.930 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-02-21T17:12:17.930 INFO:tasks.rook.operator.smithi005.stdout: osd_fsid, json.dumps(secrets)) 2022-02-21T17:12:17.930 INFO:tasks.rook.operator.smithi005.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-02-21T17:12:17.930 INFO:tasks.rook.operator.smithi005.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-02-21T17:12:17.931 INFO:tasks.rook.operator.smithi005.stdout:RuntimeError: Unable to create a new OSD id 2022-02-21T17:12:17.931 INFO:tasks.rook.operator.smithi005.stdout:: exit status 1} 2022-02-21T17:12:17.931 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:17.902207 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T17:12:18.628 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:18.624520 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:12:18.901 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:18.897719 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T17:12:21.349 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:12:22.276 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:12:22.293 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":17,"num_osds":12,"num_up_osds":1,"osd_up_since":1645463539,"num_in_osds":12,"osd_in_since":1645463533,"num_remapped_pgs":0} 2022-02-21T17:12:22.294 INFO:tasks.rook: have 1/12 OSDs 2022-02-21T17:12:32.295 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:12:32.962 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:32.957643 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T17:12:32.962 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:32.957671 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T17:12:32.962 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:32.957685 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T17:12:33.389 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:12:33.414 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"num_osds":12,"num_up_osds":2,"osd_up_since":1645463546,"num_in_osds":12,"osd_in_since":1645463533,"num_remapped_pgs":0} 2022-02-21T17:12:33.415 INFO:tasks.rook: have 2/12 OSDs 2022-02-21T17:12:33.591 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:33.588238 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:12:33.594 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:33.591160 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:12:33.597 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:33.591330 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:12:34.818 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:34.815237 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T17:12:35.108 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:35.104750 I | op-mon: start running mons 2022-02-21T17:12:35.132 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:35.128817 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:12:35.167 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:35.164233 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789","10.106.73.178:6789","10.109.91.196:6789"]}] data:a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:2] 2022-02-21T17:12:35.328 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:35.324841 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:12:35.329 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:35.325149 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:12:36.994 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:36.991106 I | op-mon: targeting the mon count 3 2022-02-21T17:12:37.001 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:36.998036 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:12:37.887 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:37.884355 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:12:37.888 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:37.884391 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:12:38.941 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:38.937821 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:12:38.941 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:38.937861 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:12:39.347 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:39.344044 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:12:39.347 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:39.344091 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:12:39.712 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:39.708659 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:12:39.712 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:39.708687 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T17:12:40.050 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:40.046782 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T17:12:40.050 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:40.046816 I | op-mon: checking for basic quorum with existing mons 2022-02-21T17:12:40.340 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:40.336948 I | op-mon: mon "b" endpoint is [v2:10.106.73.178:3300,v1:10.106.73.178:6789] 2022-02-21T17:12:41.158 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:41.155352 I | op-mon: mon "c" endpoint is [v2:10.109.91.196:3300,v1:10.109.91.196:6789] 2022-02-21T17:12:41.326 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:41.323078 I | op-mon: mon "a" endpoint is [v2:10.110.173.53:3300,v1:10.110.173.53:6789] 2022-02-21T17:12:41.362 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:41.359258 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789","10.106.73.178:6789","10.109.91.196:6789"]}] data:a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:2] 2022-02-21T17:12:41.653 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:41.650226 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:12:41.654 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:41.650520 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:12:42.074 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:42.071109 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-02-21T17:12:42.084 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:42.081416 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-02-21T17:12:42.084 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:42.081446 I | op-mon: waiting for mon quorum with [b c a] 2022-02-21T17:12:42.662 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:42.659169 I | op-mon: mons running: [b c a] 2022-02-21T17:12:43.075 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.071921 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:12:43.081 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.078339 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-02-21T17:12:43.092 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.089785 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-02-21T17:12:43.092 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.089816 I | op-mon: waiting for mon quorum with [b c a] 2022-02-21T17:12:43.277 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.274411 I | op-mon: mons running: [b c a] 2022-02-21T17:12:43.416 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:12:43.723 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.720554 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:12:43.731 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.728235 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T17:12:43.742 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.739252 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T17:12:43.742 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.739277 I | op-mon: waiting for mon quorum with [b c a] 2022-02-21T17:12:43.864 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:43.860881 I | op-mon: mons running: [b c a] 2022-02-21T17:12:43.927 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:12:43.947 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":30,"num_osds":12,"num_up_osds":8,"osd_up_since":1645463563,"num_in_osds":12,"osd_in_since":1645463533,"num_remapped_pgs":1} 2022-02-21T17:12:43.948 INFO:tasks.rook: have 8/12 OSDs 2022-02-21T17:12:44.275 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:44.271949 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:12:44.275 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:44.271977 I | op-mon: mons created: 3 2022-02-21T17:12:44.758 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:44.754830 I | op-mon: waiting for mon quorum with [b c a] 2022-02-21T17:12:44.807 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:44.803575 I | op-mon: mons running: [b c a] 2022-02-21T17:12:45.253 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:45.249976 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:12:45.253 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:45.250024 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-02-21T17:12:45.697 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:45.693805 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-02-21T17:12:46.196 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:46.193189 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-02-21T17:12:46.621 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:46.617728 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-02-21T17:12:47.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:47.095485 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-02-21T17:12:47.099 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:47.095518 I | cephclient: getting or creating ceph auth key "client.crash" 2022-02-21T17:12:47.545 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:47.542630 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-02-21T17:12:47.981 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:47.977765 I | cephclient: successfully enabled msgr2 protocol 2022-02-21T17:12:47.981 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:47.977810 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-02-21T17:12:47.982 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:47.977819 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-02-21T17:12:48.489 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:48.486462 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-02-21T17:12:48.519 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:48.516552 I | op-mgr: start running mgr 2022-02-21T17:12:48.520 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:48.516586 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-02-21T17:12:48.999 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:48.996382 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-02-21T17:12:49.007 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.004363 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-02-21T17:12:49.010 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.007725 I | op-mgr: setting services to point to mgr "a" 2022-02-21T17:12:49.125 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.121081 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-02-21T17:12:49.125 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.121193 I | op-mgr: successful modules: balancer 2022-02-21T17:12:49.137 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.134305 I | op-osd: start running osds in namespace "rook-ceph" 2022-02-21T17:12:49.137 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.134338 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-02-21T17:12:49.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.180590 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-02-21T17:12:49.324 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316175 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi005-lv-1" 2022-02-21T17:12:49.325 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316205 I | op-osd: OSD PVC "smithi005-lv-1-data-0mcn2z" already exists 2022-02-21T17:12:49.325 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316214 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi005-lv-2" 2022-02-21T17:12:49.325 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316221 I | op-osd: OSD PVC "smithi005-lv-2-data-0hnmnc" already exists 2022-02-21T17:12:49.326 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316226 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi005-lv-3" 2022-02-21T17:12:49.326 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316232 I | op-osd: OSD PVC "smithi005-lv-3-data-0glx9j" already exists 2022-02-21T17:12:49.327 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316237 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi005-lv-4" 2022-02-21T17:12:49.327 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316243 I | op-osd: OSD PVC "smithi005-lv-4-data-09b8xq" already exists 2022-02-21T17:12:49.328 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316251 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi033-lv-1" 2022-02-21T17:12:49.328 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316266 I | op-osd: OSD PVC "smithi033-lv-1-data-0bqbtn" already exists 2022-02-21T17:12:49.329 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316272 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi033-lv-2" 2022-02-21T17:12:49.329 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316278 I | op-osd: OSD PVC "smithi033-lv-2-data-0drpqd" already exists 2022-02-21T17:12:49.330 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316284 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi033-lv-3" 2022-02-21T17:12:49.330 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316298 I | op-osd: OSD PVC "smithi033-lv-3-data-0t898k" already exists 2022-02-21T17:12:49.331 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316307 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi033-lv-4" 2022-02-21T17:12:49.331 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316318 I | op-osd: OSD PVC "smithi033-lv-4-data-0p96d6" already exists 2022-02-21T17:12:49.332 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316326 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi067-lv-1" 2022-02-21T17:12:49.332 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316338 I | op-osd: OSD PVC "smithi067-lv-1-data-05cnzw" already exists 2022-02-21T17:12:49.332 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316350 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi067-lv-2" 2022-02-21T17:12:49.333 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316364 I | op-osd: OSD PVC "smithi067-lv-2-data-08phkg" already exists 2022-02-21T17:12:49.333 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316374 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi067-lv-3" 2022-02-21T17:12:49.334 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316387 I | op-osd: OSD PVC "smithi067-lv-3-data-0d6rns" already exists 2022-02-21T17:12:49.334 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316403 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi067-lv-4" 2022-02-21T17:12:49.335 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.316411 I | op-osd: OSD PVC "smithi067-lv-4-data-0kh9ql" already exists 2022-02-21T17:12:49.375 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370342 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-1-data-0mcn2z" 2022-02-21T17:12:49.376 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370362 I | op-osd: skipping OSD prepare job creation for PVC "smithi005-lv-1-data-0mcn2z" because OSD daemon using the PVC already exists 2022-02-21T17:12:49.376 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370366 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:12:49.377 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370369 I | op-osd: skipping OSD prepare job creation for PVC "smithi005-lv-2-data-0hnmnc" because OSD daemon using the PVC already exists 2022-02-21T17:12:49.377 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370372 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-3-data-0glx9j" 2022-02-21T17:12:49.377 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370375 I | op-osd: skipping OSD prepare job creation for PVC "smithi005-lv-3-data-0glx9j" because OSD daemon using the PVC already exists 2022-02-21T17:12:49.378 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370380 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-4-data-09b8xq" 2022-02-21T17:12:49.379 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370385 I | op-osd: skipping OSD prepare job creation for PVC "smithi005-lv-4-data-09b8xq" because OSD daemon using the PVC already exists 2022-02-21T17:12:49.379 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370389 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:12:49.380 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370392 I | op-osd: skipping OSD prepare job creation for PVC "smithi033-lv-1-data-0bqbtn" because OSD daemon using the PVC already exists 2022-02-21T17:12:49.380 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370395 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-2-data-0drpqd" 2022-02-21T17:12:49.381 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370398 I | op-osd: skipping OSD prepare job creation for PVC "smithi033-lv-2-data-0drpqd" because OSD daemon using the PVC already exists 2022-02-21T17:12:49.381 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370401 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-3-data-0t898k" 2022-02-21T17:12:49.382 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370404 I | op-osd: skipping OSD prepare job creation for PVC "smithi033-lv-3-data-0t898k" because OSD daemon using the PVC already exists 2022-02-21T17:12:49.382 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.370408 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-4-data-0p96d6" 2022-02-21T17:12:49.697 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.692335 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-smithi033-lv-4-data-0p96d6 to start a new one 2022-02-21T17:12:49.747 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:49.716371 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi033-lv-4-data-0p96d6 still exists 2022-02-21T17:12:50.639 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:50.635927 I | op-mgr: successful modules: mgr module(s) from the spec 2022-02-21T17:12:50.644 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:50.641188 I | op-mgr: successful modules: dashboard 2022-02-21T17:12:50.646 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:50.641536 I | op-mgr: successful modules: prometheus 2022-02-21T17:12:52.415 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.412161 I | op-mgr: successful modules: orchestrator modules 2022-02-21T17:12:52.723 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.720624 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi033-lv-4-data-0p96d6 deleted 2022-02-21T17:12:52.732 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728256 I | op-osd: started OSD provisioning job for PVC "smithi033-lv-4-data-0p96d6" 2022-02-21T17:12:52.732 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728287 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-1-data-05cnzw" 2022-02-21T17:12:52.732 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728296 I | op-osd: skipping OSD prepare job creation for PVC "smithi067-lv-1-data-05cnzw" because OSD daemon using the PVC already exists 2022-02-21T17:12:52.733 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728304 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-2-data-08phkg" 2022-02-21T17:12:52.733 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728311 I | op-osd: skipping OSD prepare job creation for PVC "smithi067-lv-2-data-08phkg" because OSD daemon using the PVC already exists 2022-02-21T17:12:52.733 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728317 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-3-data-0d6rns" 2022-02-21T17:12:52.733 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728324 I | op-osd: skipping OSD prepare job creation for PVC "smithi067-lv-3-data-0d6rns" because OSD daemon using the PVC already exists 2022-02-21T17:12:52.734 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728331 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:12:52.734 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728341 I | op-osd: skipping OSD prepare job creation for PVC "smithi067-lv-4-data-0kh9ql" because OSD daemon using the PVC already exists 2022-02-21T17:12:52.734 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728351 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-02-21T17:12:52.734 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.728360 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-02-21T17:12:52.756 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.753090 I | op-osd: OSD orchestration status for PVC smithi033-lv-4-data-0p96d6 is "starting" 2022-02-21T17:12:52.857 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.854478 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:52.874 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.871734 I | op-osd: updating OSD 0 on PVC "smithi005-lv-1-data-0mcn2z" 2022-02-21T17:12:52.875 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:52.871764 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-1-data-0mcn2z" 2022-02-21T17:12:53.138 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:53.135721 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:53.150 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:53.147559 I | op-osd: updating OSD 1 on PVC "smithi067-lv-3-data-0d6rns" 2022-02-21T17:12:53.150 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:53.147579 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-3-data-0d6rns" 2022-02-21T17:12:53.401 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:53.398669 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:53.411 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:53.408647 I | op-osd: updating OSD 10 on PVC "smithi005-lv-4-data-09b8xq" 2022-02-21T17:12:53.412 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:53.408690 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-4-data-09b8xq" 2022-02-21T17:12:53.571 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:53.568039 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:53.585 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:53.582168 I | op-osd: updating OSD 2 on PVC "smithi067-lv-1-data-05cnzw" 2022-02-21T17:12:53.585 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:53.582203 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-1-data-05cnzw" 2022-02-21T17:12:53.949 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:12:54.171 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:54.168183 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:54.265 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:54.262288 I | op-osd: updating OSD 3 on PVC "smithi005-lv-3-data-0glx9j" 2022-02-21T17:12:54.265 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:54.262312 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-3-data-0glx9j" 2022-02-21T17:12:54.466 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:12:54.486 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":36,"num_osds":12,"num_up_osds":11,"osd_up_since":1645463568,"num_in_osds":12,"osd_in_since":1645463533,"num_remapped_pgs":0} 2022-02-21T17:12:54.487 INFO:tasks.rook: have 11/12 OSDs 2022-02-21T17:12:54.971 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:54.968193 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:55.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:55.061865 I | op-osd: updating OSD 4 on PVC "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:12:55.065 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:55.061899 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:12:55.770 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:55.767305 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:55.864 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:55.861212 I | op-osd: updating OSD 5 on PVC "smithi067-lv-2-data-08phkg" 2022-02-21T17:12:55.864 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:55.861248 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-2-data-08phkg" 2022-02-21T17:12:56.571 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:56.568792 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:56.690 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:56.686834 I | op-osd: updating OSD 6 on PVC "smithi033-lv-3-data-0t898k" 2022-02-21T17:12:56.691 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:56.686871 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-3-data-0t898k" 2022-02-21T17:12:57.370 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:57.367690 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:57.465 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:57.462031 I | op-osd: updating OSD 7 on PVC "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:12:57.465 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:57.462065 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:12:58.172 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:58.169704 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:58.301 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:58.298207 I | op-osd: updating OSD 8 on PVC "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:12:58.301 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:58.298247 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:12:58.973 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:58.970851 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:12:59.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:59.061339 I | op-osd: updating OSD 9 on PVC "smithi033-lv-2-data-0drpqd" 2022-02-21T17:12:59.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:12:59.061382 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-2-data-0drpqd" 2022-02-21T17:13:00.605 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:00.602939 I | op-osd: OSD orchestration status for node smithi033-lv-4-data-0p96d6 is "orchestrating" 2022-02-21T17:13:00.661 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:00.659004 I | op-osd: OSD orchestration status for PVC smithi033-lv-4-data-0p96d6 is "orchestrating" 2022-02-21T17:13:01.816 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:01.813272 I | op-osd: OSD orchestration status for PVC smithi033-lv-4-data-0p96d6 is "completed" 2022-02-21T17:13:01.816 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:01.813309 I | op-osd: creating OSD 11 on PVC "smithi033-lv-4-data-0p96d6" 2022-02-21T17:13:01.816 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:01.813324 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-4-data-0p96d6" 2022-02-21T17:13:02.429 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:02.425850 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-02-21T17:13:02.429 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:02.425887 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T17:13:04.488 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:13:05.081 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:05.102 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":36,"num_osds":12,"num_up_osds":11,"osd_up_since":1645463568,"num_in_osds":12,"osd_in_since":1645463533,"num_remapped_pgs":0} 2022-02-21T17:13:05.103 INFO:tasks.rook: have 11/12 OSDs 2022-02-21T17:13:15.104 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:13:15.609 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:15.632 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":36,"num_osds":12,"num_up_osds":11,"osd_up_since":1645463568,"num_in_osds":12,"osd_in_since":1645463533,"num_remapped_pgs":0} 2022-02-21T17:13:15.633 INFO:tasks.rook: have 11/12 OSDs 2022-02-21T17:13:25.635 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd stat -f json 2022-02-21T17:13:26.150 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:26.170 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":43,"num_osds":12,"num_up_osds":12,"osd_up_since":1645463600,"num_in_osds":12,"osd_in_since":1645463533,"num_remapped_pgs":0} 2022-02-21T17:13:26.171 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-02-21T17:13:26.171 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- cat /etc/ceph/ceph.conf 2022-02-21T17:13:26.345 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2022-02-21T17:13:26.345 INFO:teuthology.orchestra.run.smithi005.stdout:mon_host = 10.110.173.53:6789,10.106.73.178:6789,10.109.91.196:6789 2022-02-21T17:13:26.346 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:26.346 INFO:teuthology.orchestra.run.smithi005.stdout:[client.admin] 2022-02-21T17:13:26.346 INFO:teuthology.orchestra.run.smithi005.stdout:keyring = /etc/ceph/keyring 2022-02-21T17:13:26.350 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- cat /etc/ceph/keyring 2022-02-21T17:13:26.545 INFO:teuthology.orchestra.run.smithi005.stdout:[client.admin] 2022-02-21T17:13:26.546 INFO:teuthology.orchestra.run.smithi005.stdout:key = AQD2xhNi6QDXBxAAvMxTuGsq+2ohyH+2Mf1psw== 2022-02-21T17:13:26.550 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph 2022-02-21T17:13:26.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph 2022-02-21T17:13:26.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2022-02-21T17:13:26.599 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:13:26.599 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.conf 2022-02-21T17:13:26.615 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:13:26.616 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/keyring 2022-02-21T17:13:26.629 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:13:26.629 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.conf 2022-02-21T17:13:26.644 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:13:26.644 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/keyring 2022-02-21T17:13:26.655 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:13:26.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.conf 2022-02-21T17:13:26.672 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:13:26.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/keyring 2022-02-21T17:13:26.732 INFO:tasks.rook:Setting up client nodes... 2022-02-21T17:13:26.733 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-02-21T17:13:27.430 INFO:teuthology.orchestra.run.smithi005.stdout:[client.a] 2022-02-21T17:13:27.430 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA3yBNi0AJDFBAAw1/pxwcN88u4Tdk/B4Adlw== 2022-02-21T17:13:27.452 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-02-21T17:13:27.452 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-02-21T17:13:27.452 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-02-21T17:13:27.479 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-02-21T17:13:28.223 INFO:teuthology.orchestra.run.smithi005.stdout:[client.b] 2022-02-21T17:13:28.223 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA4yBNieyRXCBAAI4SiOtcamvgKgnkrdHnlQw== 2022-02-21T17:13:28.251 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-02-21T17:13:28.251 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-02-21T17:13:28.251 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-02-21T17:13:28.275 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-02-21T17:13:29.048 INFO:teuthology.orchestra.run.smithi005.stdout:[client.c] 2022-02-21T17:13:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA4yBNiRv9rNxAAjcg1QVSFy/wOcPZpfLgmZw== 2022-02-21T17:13:29.072 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2022-02-21T17:13:29.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-02-21T17:13:29.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-02-21T17:13:29.098 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd dump --format=json 2022-02-21T17:13:29.618 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:29.619 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":43,"fsid":"40f92d2b-27d2-4f26-ad3f-8c38961d94a2","created":"2022-02-21T17:08:24.862435+0000","modified":"2022-02-21T17:13:25.661298+0000","last_up_change":"2022-02-21T17:13:20.833672+0000","last_in_change":"2022-02-21T17:12:13.760882+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-02-21T17:12:37.863239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7b8d33f3-a5c1-43b8-8310-27b6306fa366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6800","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6801","nonce":2358489059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6802","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6803","nonce":2358489059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6806","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6807","nonce":2358489059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6804","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6805","nonce":2358489059}]},"public_addr":"10.248.32.11:6801/2358489059","cluster_addr":"10.248.32.11:6803/2358489059","heartbeat_back_addr":"10.248.32.11:6807/2358489059","heartbeat_front_addr":"10.248.32.11:6805/2358489059","state":["exists","up"]},{"osd":1,"uuid":"e0debea9-86a7-468f-b732-acea8778e8db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6800","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6801","nonce":3783421224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6802","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6803","nonce":3783421224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6806","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6807","nonce":3783421224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6804","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6805","nonce":3783421224}]},"public_addr":"10.248.34.15:6801/3783421224","cluster_addr":"10.248.34.15:6803/3783421224","heartbeat_back_addr":"10.248.34.15:6807/3783421224","heartbeat_front_addr":"10.248.34.15:6805/3783421224","state":["exists","up"]},{"osd":2,"uuid":"3ad1756f-42c2-40ca-a637-c23e7bdac26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6800","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6801","nonce":3074650996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6802","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6803","nonce":3074650996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6806","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6807","nonce":3074650996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6804","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6805","nonce":3074650996}]},"public_addr":"10.248.34.14:6801/3074650996","cluster_addr":"10.248.34.14:6803/3074650996","heartbeat_back_addr":"10.248.34.14:6807/3074650996","heartbeat_front_addr":"10.248.34.14:6805/3074650996","state":["exists","up"]},{"osd":3,"uuid":"f20c8870-23ce-4099-ae39-a740f7741af9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6800","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6801","nonce":3989780310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6802","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6803","nonce":3989780310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6806","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6807","nonce":3989780310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6804","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6805","nonce":3989780310}]},"public_addr":"10.248.32.12:6801/3989780310","cluster_addr":"10.248.32.12:6803/3989780310","heartbeat_back_addr":"10.248.32.12:6807/3989780310","heartbeat_front_addr":"10.248.32.12:6805/3989780310","state":["exists","up"]},{"osd":4,"uuid":"2b2f900b-9acb-4b78-9a8a-4b93cd43f25e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6800","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6801","nonce":3013841693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6802","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6803","nonce":3013841693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6806","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6807","nonce":3013841693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6804","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6805","nonce":3013841693}]},"public_addr":"10.248.32.13:6801/3013841693","cluster_addr":"10.248.32.13:6803/3013841693","heartbeat_back_addr":"10.248.32.13:6807/3013841693","heartbeat_front_addr":"10.248.32.13:6805/3013841693","state":["exists","up"]},{"osd":5,"uuid":"5d234e23-7ffe-426d-95f9-e2056584ad81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6800","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6801","nonce":155899310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6802","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6803","nonce":155899310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6806","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6807","nonce":155899310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6804","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6805","nonce":155899310}]},"public_addr":"10.248.34.16:6801/155899310","cluster_addr":"10.248.34.16:6803/155899310","heartbeat_back_addr":"10.248.34.16:6807/155899310","heartbeat_front_addr":"10.248.34.16:6805/155899310","state":["exists","up"]},{"osd":6,"uuid":"9f99b4b4-ba96-41ba-be6b-73b9389b5190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6800","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6801","nonce":2518234005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6802","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6803","nonce":2518234005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6806","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6807","nonce":2518234005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6804","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6805","nonce":2518234005}]},"public_addr":"10.248.33.15:6801/2518234005","cluster_addr":"10.248.33.15:6803/2518234005","heartbeat_back_addr":"10.248.33.15:6807/2518234005","heartbeat_front_addr":"10.248.33.15:6805/2518234005","state":["exists","up"]},{"osd":7,"uuid":"5df9018b-adaa-42ec-8b12-8ae1dceb88f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6800","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6801","nonce":3455698330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6802","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6803","nonce":3455698330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6806","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6807","nonce":3455698330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6804","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6805","nonce":3455698330}]},"public_addr":"10.248.33.17:6801/3455698330","cluster_addr":"10.248.33.17:6803/3455698330","heartbeat_back_addr":"10.248.33.17:6807/3455698330","heartbeat_front_addr":"10.248.33.17:6805/3455698330","state":["exists","up"]},{"osd":8,"uuid":"8ccf2c62-a638-40bf-b0fd-a2376d1e5d98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6800","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6801","nonce":3143917843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6802","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6803","nonce":3143917843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6806","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6807","nonce":3143917843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6804","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6805","nonce":3143917843}]},"public_addr":"10.248.34.17:6801/3143917843","cluster_addr":"10.248.34.17:6803/3143917843","heartbeat_back_addr":"10.248.34.17:6807/3143917843","heartbeat_front_addr":"10.248.34.17:6805/3143917843","state":["exists","up"]},{"osd":9,"uuid":"49e1069e-fadb-4eb2-98d9-05c466684e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6800","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6801","nonce":3101896865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6802","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6803","nonce":3101896865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6806","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6807","nonce":3101896865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6804","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6805","nonce":3101896865}]},"public_addr":"10.248.33.18:6801/3101896865","cluster_addr":"10.248.33.18:6803/3101896865","heartbeat_back_addr":"10.248.33.18:6807/3101896865","heartbeat_front_addr":"10.248.33.18:6805/3101896865","state":["exists","up"]},{"osd":10,"uuid":"81272f37-208a-47b8-99a9-7485058bc4d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6800","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6801","nonce":1139102259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6802","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6803","nonce":1139102259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6806","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6807","nonce":1139102259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6804","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6805","nonce":1139102259}]},"public_addr":"10.248.32.14:6801/1139102259","cluster_addr":"10.248.32.14:6803/1139102259","heartbeat_back_addr":"10.248.32.14:6807/1139102259","heartbeat_front_addr":"10.248.32.14:6805/1139102259","state":["exists","up"]},{"osd":11,"uuid":"2f54b59c-2059-4f2f-866b-5f1198aca97d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6800","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6801","nonce":2076908842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6802","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6803","nonce":2076908842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6806","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6807","nonce":2076908842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6804","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6805","nonce":2076908842}]},"public_addr":"10.248.33.20:6801/2076908842","cluster_addr":"10.248.33.20:6803/2076908842","heartbeat_back_addr":"10.248.33.20:6807/2076908842","heartbeat_front_addr":"10.248.33.20:6805/2076908842","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:30.086368+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:21.073186+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:16.106254+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:34.997662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:33.697624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:31.452941+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:41.033453+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:44.100724+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:33.919978+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:44.176659+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:46.029380+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:13:19.081544+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.33.8:0/1700670713":"2022-02-22T17:11:02.685558+0000","10.248.33.8:0/1109669853":"2022-02-22T17:11:02.685558+0000","10.248.33.8:6801/1361989934":"2022-02-22T17:11:02.685558+0000","10.248.33.8:0/261868923":"2022-02-22T17:11:02.685558+0000","10.248.33.8:6800/1361989934":"2022-02-22T17:11:02.685558+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-02-21T17:13:29.649 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-02-21T17:12:37.863239+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-02-21T17:13:29.650 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd pool get .mgr pg_num 2022-02-21T17:13:30.199 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 1 2022-02-21T17:13:30.219 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-02-21T17:13:30.220 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-02-21T17:13:30.221 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph mgr dump --format=json 2022-02-21T17:13:30.786 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:30.807 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":14,"active_gid":4350,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.8:6800","nonce":3939981250},{"type":"v1","addr":"10.248.33.8:6801","nonce":3939981250}]},"active_addr":"10.248.33.8:6801/3939981250","active_change":"2022-02-21T17:11:02.717184+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.248.33.8:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.248.33.8:0","nonce":2093455966}]},{"addrvec":[{"type":"v2","addr":"10.248.33.8:0","nonce":4275757546}]},{"addrvec":[{"type":"v2","addr":"10.248.33.8:0","nonce":714071810}]},{"addrvec":[{"type":"v2","addr":"10.248.33.8:0","nonce":2808554002}]}]}} 2022-02-21T17:13:30.810 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-02-21T17:13:30.810 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-02-21T17:13:30.811 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd dump --format=json 2022-02-21T17:13:31.347 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:31.348 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":43,"fsid":"40f92d2b-27d2-4f26-ad3f-8c38961d94a2","created":"2022-02-21T17:08:24.862435+0000","modified":"2022-02-21T17:13:25.661298+0000","last_up_change":"2022-02-21T17:13:20.833672+0000","last_in_change":"2022-02-21T17:12:13.760882+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-02-21T17:12:37.863239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7b8d33f3-a5c1-43b8-8310-27b6306fa366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6800","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6801","nonce":2358489059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6802","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6803","nonce":2358489059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6806","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6807","nonce":2358489059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6804","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6805","nonce":2358489059}]},"public_addr":"10.248.32.11:6801/2358489059","cluster_addr":"10.248.32.11:6803/2358489059","heartbeat_back_addr":"10.248.32.11:6807/2358489059","heartbeat_front_addr":"10.248.32.11:6805/2358489059","state":["exists","up"]},{"osd":1,"uuid":"e0debea9-86a7-468f-b732-acea8778e8db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6800","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6801","nonce":3783421224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6802","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6803","nonce":3783421224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6806","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6807","nonce":3783421224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6804","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6805","nonce":3783421224}]},"public_addr":"10.248.34.15:6801/3783421224","cluster_addr":"10.248.34.15:6803/3783421224","heartbeat_back_addr":"10.248.34.15:6807/3783421224","heartbeat_front_addr":"10.248.34.15:6805/3783421224","state":["exists","up"]},{"osd":2,"uuid":"3ad1756f-42c2-40ca-a637-c23e7bdac26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6800","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6801","nonce":3074650996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6802","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6803","nonce":3074650996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6806","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6807","nonce":3074650996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6804","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6805","nonce":3074650996}]},"public_addr":"10.248.34.14:6801/3074650996","cluster_addr":"10.248.34.14:6803/3074650996","heartbeat_back_addr":"10.248.34.14:6807/3074650996","heartbeat_front_addr":"10.248.34.14:6805/3074650996","state":["exists","up"]},{"osd":3,"uuid":"f20c8870-23ce-4099-ae39-a740f7741af9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6800","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6801","nonce":3989780310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6802","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6803","nonce":3989780310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6806","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6807","nonce":3989780310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6804","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6805","nonce":3989780310}]},"public_addr":"10.248.32.12:6801/3989780310","cluster_addr":"10.248.32.12:6803/3989780310","heartbeat_back_addr":"10.248.32.12:6807/3989780310","heartbeat_front_addr":"10.248.32.12:6805/3989780310","state":["exists","up"]},{"osd":4,"uuid":"2b2f900b-9acb-4b78-9a8a-4b93cd43f25e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6800","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6801","nonce":3013841693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6802","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6803","nonce":3013841693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6806","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6807","nonce":3013841693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6804","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6805","nonce":3013841693}]},"public_addr":"10.248.32.13:6801/3013841693","cluster_addr":"10.248.32.13:6803/3013841693","heartbeat_back_addr":"10.248.32.13:6807/3013841693","heartbeat_front_addr":"10.248.32.13:6805/3013841693","state":["exists","up"]},{"osd":5,"uuid":"5d234e23-7ffe-426d-95f9-e2056584ad81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6800","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6801","nonce":155899310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6802","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6803","nonce":155899310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6806","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6807","nonce":155899310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6804","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6805","nonce":155899310}]},"public_addr":"10.248.34.16:6801/155899310","cluster_addr":"10.248.34.16:6803/155899310","heartbeat_back_addr":"10.248.34.16:6807/155899310","heartbeat_front_addr":"10.248.34.16:6805/155899310","state":["exists","up"]},{"osd":6,"uuid":"9f99b4b4-ba96-41ba-be6b-73b9389b5190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6800","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6801","nonce":2518234005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6802","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6803","nonce":2518234005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6806","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6807","nonce":2518234005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6804","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6805","nonce":2518234005}]},"public_addr":"10.248.33.15:6801/2518234005","cluster_addr":"10.248.33.15:6803/2518234005","heartbeat_back_addr":"10.248.33.15:6807/2518234005","heartbeat_front_addr":"10.248.33.15:6805/2518234005","state":["exists","up"]},{"osd":7,"uuid":"5df9018b-adaa-42ec-8b12-8ae1dceb88f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6800","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6801","nonce":3455698330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6802","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6803","nonce":3455698330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6806","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6807","nonce":3455698330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6804","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6805","nonce":3455698330}]},"public_addr":"10.248.33.17:6801/3455698330","cluster_addr":"10.248.33.17:6803/3455698330","heartbeat_back_addr":"10.248.33.17:6807/3455698330","heartbeat_front_addr":"10.248.33.17:6805/3455698330","state":["exists","up"]},{"osd":8,"uuid":"8ccf2c62-a638-40bf-b0fd-a2376d1e5d98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6800","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6801","nonce":3143917843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6802","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6803","nonce":3143917843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6806","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6807","nonce":3143917843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6804","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6805","nonce":3143917843}]},"public_addr":"10.248.34.17:6801/3143917843","cluster_addr":"10.248.34.17:6803/3143917843","heartbeat_back_addr":"10.248.34.17:6807/3143917843","heartbeat_front_addr":"10.248.34.17:6805/3143917843","state":["exists","up"]},{"osd":9,"uuid":"49e1069e-fadb-4eb2-98d9-05c466684e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6800","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6801","nonce":3101896865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6802","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6803","nonce":3101896865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6806","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6807","nonce":3101896865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6804","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6805","nonce":3101896865}]},"public_addr":"10.248.33.18:6801/3101896865","cluster_addr":"10.248.33.18:6803/3101896865","heartbeat_back_addr":"10.248.33.18:6807/3101896865","heartbeat_front_addr":"10.248.33.18:6805/3101896865","state":["exists","up"]},{"osd":10,"uuid":"81272f37-208a-47b8-99a9-7485058bc4d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6800","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6801","nonce":1139102259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6802","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6803","nonce":1139102259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6806","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6807","nonce":1139102259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6804","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6805","nonce":1139102259}]},"public_addr":"10.248.32.14:6801/1139102259","cluster_addr":"10.248.32.14:6803/1139102259","heartbeat_back_addr":"10.248.32.14:6807/1139102259","heartbeat_front_addr":"10.248.32.14:6805/1139102259","state":["exists","up"]},{"osd":11,"uuid":"2f54b59c-2059-4f2f-866b-5f1198aca97d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6800","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6801","nonce":2076908842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6802","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6803","nonce":2076908842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6806","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6807","nonce":2076908842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6804","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6805","nonce":2076908842}]},"public_addr":"10.248.33.20:6801/2076908842","cluster_addr":"10.248.33.20:6803/2076908842","heartbeat_back_addr":"10.248.33.20:6807/2076908842","heartbeat_front_addr":"10.248.33.20:6805/2076908842","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:30.086368+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:21.073186+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:16.106254+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:34.997662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:33.697624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:31.452941+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:41.033453+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:44.100724+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:33.919978+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:44.176659+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:46.029380+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:13:19.081544+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.33.8:0/1700670713":"2022-02-22T17:11:02.685558+0000","10.248.33.8:0/1109669853":"2022-02-22T17:11:02.685558+0000","10.248.33.8:6801/1361989934":"2022-02-22T17:11:02.685558+0000","10.248.33.8:0/261868923":"2022-02-22T17:11:02.685558+0000","10.248.33.8:6800/1361989934":"2022-02-22T17:11:02.685558+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-02-21T17:13:31.366 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-02-21T17:13:31.366 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd dump --format=json 2022-02-21T17:13:31.912 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:31.912 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":43,"fsid":"40f92d2b-27d2-4f26-ad3f-8c38961d94a2","created":"2022-02-21T17:08:24.862435+0000","modified":"2022-02-21T17:13:25.661298+0000","last_up_change":"2022-02-21T17:13:20.833672+0000","last_in_change":"2022-02-21T17:12:13.760882+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-02-21T17:12:37.863239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7b8d33f3-a5c1-43b8-8310-27b6306fa366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6800","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6801","nonce":2358489059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6802","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6803","nonce":2358489059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6806","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6807","nonce":2358489059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.11:6804","nonce":2358489059},{"type":"v1","addr":"10.248.32.11:6805","nonce":2358489059}]},"public_addr":"10.248.32.11:6801/2358489059","cluster_addr":"10.248.32.11:6803/2358489059","heartbeat_back_addr":"10.248.32.11:6807/2358489059","heartbeat_front_addr":"10.248.32.11:6805/2358489059","state":["exists","up"]},{"osd":1,"uuid":"e0debea9-86a7-468f-b732-acea8778e8db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6800","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6801","nonce":3783421224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6802","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6803","nonce":3783421224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6806","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6807","nonce":3783421224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.15:6804","nonce":3783421224},{"type":"v1","addr":"10.248.34.15:6805","nonce":3783421224}]},"public_addr":"10.248.34.15:6801/3783421224","cluster_addr":"10.248.34.15:6803/3783421224","heartbeat_back_addr":"10.248.34.15:6807/3783421224","heartbeat_front_addr":"10.248.34.15:6805/3783421224","state":["exists","up"]},{"osd":2,"uuid":"3ad1756f-42c2-40ca-a637-c23e7bdac26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6800","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6801","nonce":3074650996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6802","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6803","nonce":3074650996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6806","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6807","nonce":3074650996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.14:6804","nonce":3074650996},{"type":"v1","addr":"10.248.34.14:6805","nonce":3074650996}]},"public_addr":"10.248.34.14:6801/3074650996","cluster_addr":"10.248.34.14:6803/3074650996","heartbeat_back_addr":"10.248.34.14:6807/3074650996","heartbeat_front_addr":"10.248.34.14:6805/3074650996","state":["exists","up"]},{"osd":3,"uuid":"f20c8870-23ce-4099-ae39-a740f7741af9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6800","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6801","nonce":3989780310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6802","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6803","nonce":3989780310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6806","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6807","nonce":3989780310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.12:6804","nonce":3989780310},{"type":"v1","addr":"10.248.32.12:6805","nonce":3989780310}]},"public_addr":"10.248.32.12:6801/3989780310","cluster_addr":"10.248.32.12:6803/3989780310","heartbeat_back_addr":"10.248.32.12:6807/3989780310","heartbeat_front_addr":"10.248.32.12:6805/3989780310","state":["exists","up"]},{"osd":4,"uuid":"2b2f900b-9acb-4b78-9a8a-4b93cd43f25e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6800","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6801","nonce":3013841693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6802","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6803","nonce":3013841693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6806","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6807","nonce":3013841693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.13:6804","nonce":3013841693},{"type":"v1","addr":"10.248.32.13:6805","nonce":3013841693}]},"public_addr":"10.248.32.13:6801/3013841693","cluster_addr":"10.248.32.13:6803/3013841693","heartbeat_back_addr":"10.248.32.13:6807/3013841693","heartbeat_front_addr":"10.248.32.13:6805/3013841693","state":["exists","up"]},{"osd":5,"uuid":"5d234e23-7ffe-426d-95f9-e2056584ad81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6800","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6801","nonce":155899310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6802","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6803","nonce":155899310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6806","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6807","nonce":155899310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.16:6804","nonce":155899310},{"type":"v1","addr":"10.248.34.16:6805","nonce":155899310}]},"public_addr":"10.248.34.16:6801/155899310","cluster_addr":"10.248.34.16:6803/155899310","heartbeat_back_addr":"10.248.34.16:6807/155899310","heartbeat_front_addr":"10.248.34.16:6805/155899310","state":["exists","up"]},{"osd":6,"uuid":"9f99b4b4-ba96-41ba-be6b-73b9389b5190","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6800","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6801","nonce":2518234005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6802","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6803","nonce":2518234005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6806","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6807","nonce":2518234005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.15:6804","nonce":2518234005},{"type":"v1","addr":"10.248.33.15:6805","nonce":2518234005}]},"public_addr":"10.248.33.15:6801/2518234005","cluster_addr":"10.248.33.15:6803/2518234005","heartbeat_back_addr":"10.248.33.15:6807/2518234005","heartbeat_front_addr":"10.248.33.15:6805/2518234005","state":["exists","up"]},{"osd":7,"uuid":"5df9018b-adaa-42ec-8b12-8ae1dceb88f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6800","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6801","nonce":3455698330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6802","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6803","nonce":3455698330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6806","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6807","nonce":3455698330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.17:6804","nonce":3455698330},{"type":"v1","addr":"10.248.33.17:6805","nonce":3455698330}]},"public_addr":"10.248.33.17:6801/3455698330","cluster_addr":"10.248.33.17:6803/3455698330","heartbeat_back_addr":"10.248.33.17:6807/3455698330","heartbeat_front_addr":"10.248.33.17:6805/3455698330","state":["exists","up"]},{"osd":8,"uuid":"8ccf2c62-a638-40bf-b0fd-a2376d1e5d98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6800","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6801","nonce":3143917843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6802","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6803","nonce":3143917843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6806","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6807","nonce":3143917843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.34.17:6804","nonce":3143917843},{"type":"v1","addr":"10.248.34.17:6805","nonce":3143917843}]},"public_addr":"10.248.34.17:6801/3143917843","cluster_addr":"10.248.34.17:6803/3143917843","heartbeat_back_addr":"10.248.34.17:6807/3143917843","heartbeat_front_addr":"10.248.34.17:6805/3143917843","state":["exists","up"]},{"osd":9,"uuid":"49e1069e-fadb-4eb2-98d9-05c466684e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6800","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6801","nonce":3101896865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6802","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6803","nonce":3101896865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6806","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6807","nonce":3101896865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.18:6804","nonce":3101896865},{"type":"v1","addr":"10.248.33.18:6805","nonce":3101896865}]},"public_addr":"10.248.33.18:6801/3101896865","cluster_addr":"10.248.33.18:6803/3101896865","heartbeat_back_addr":"10.248.33.18:6807/3101896865","heartbeat_front_addr":"10.248.33.18:6805/3101896865","state":["exists","up"]},{"osd":10,"uuid":"81272f37-208a-47b8-99a9-7485058bc4d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6800","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6801","nonce":1139102259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6802","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6803","nonce":1139102259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6806","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6807","nonce":1139102259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.32.14:6804","nonce":1139102259},{"type":"v1","addr":"10.248.32.14:6805","nonce":1139102259}]},"public_addr":"10.248.32.14:6801/1139102259","cluster_addr":"10.248.32.14:6803/1139102259","heartbeat_back_addr":"10.248.32.14:6807/1139102259","heartbeat_front_addr":"10.248.32.14:6805/1139102259","state":["exists","up"]},{"osd":11,"uuid":"2f54b59c-2059-4f2f-866b-5f1198aca97d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6800","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6801","nonce":2076908842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6802","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6803","nonce":2076908842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6806","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6807","nonce":2076908842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.33.20:6804","nonce":2076908842},{"type":"v1","addr":"10.248.33.20:6805","nonce":2076908842}]},"public_addr":"10.248.33.20:6801/2076908842","cluster_addr":"10.248.33.20:6803/2076908842","heartbeat_back_addr":"10.248.33.20:6807/2076908842","heartbeat_front_addr":"10.248.33.20:6805/2076908842","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:30.086368+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:21.073186+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:16.106254+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:34.997662+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:33.697624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:31.452941+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:41.033453+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:44.100724+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:33.919978+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:44.176659+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:12:46.029380+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-02-21T17:13:19.081544+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.33.8:0/1700670713":"2022-02-22T17:11:02.685558+0000","10.248.33.8:0/1109669853":"2022-02-22T17:11:02.685558+0000","10.248.33.8:6801/1361989934":"2022-02-22T17:11:02.685558+0000","10.248.33.8:0/261868923":"2022-02-22T17:11:02.685558+0000","10.248.33.8:6800/1361989934":"2022-02-22T17:11:02.685558+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-02-21T17:13:31.932 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.0 flush_pg_stats 2022-02-21T17:13:31.933 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.1 flush_pg_stats 2022-02-21T17:13:31.933 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.2 flush_pg_stats 2022-02-21T17:13:31.934 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.3 flush_pg_stats 2022-02-21T17:13:31.935 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.4 flush_pg_stats 2022-02-21T17:13:31.936 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.5 flush_pg_stats 2022-02-21T17:13:31.936 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.6 flush_pg_stats 2022-02-21T17:13:31.937 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.7 flush_pg_stats 2022-02-21T17:13:31.938 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.8 flush_pg_stats 2022-02-21T17:13:31.938 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.9 flush_pg_stats 2022-02-21T17:13:31.939 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.10 flush_pg_stats 2022-02-21T17:13:31.940 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph tell osd.11 flush_pg_stats 2022-02-21T17:13:32.491 INFO:teuthology.orchestra.run.smithi005.stdout:137438953484 2022-02-21T17:13:32.492 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.9 2022-02-21T17:13:32.724 INFO:teuthology.orchestra.run.smithi005.stdout:111669149710 2022-02-21T17:13:32.724 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.0 2022-02-21T17:13:32.849 INFO:teuthology.orchestra.run.smithi005.stdout:103079215117 2022-02-21T17:13:32.849 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.5 2022-02-21T17:13:32.855 INFO:teuthology.orchestra.run.smithi005.stdout:111669149709 2022-02-21T17:13:32.855 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.4 2022-02-21T17:13:32.865 INFO:teuthology.orchestra.run.smithi005.stdout:73014444050 2022-02-21T17:13:32.865 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.2 2022-02-21T17:13:32.875 INFO:teuthology.orchestra.run.smithi005.stdout:128849018893 2022-02-21T17:13:32.876 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.6 2022-02-21T17:13:32.928 INFO:teuthology.orchestra.run.smithi005.stdout:167503724550 2022-02-21T17:13:32.929 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.11 2022-02-21T17:13:32.935 INFO:teuthology.orchestra.run.smithi005.stdout:85899345936 2022-02-21T17:13:32.936 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.1 2022-02-21T17:13:32.945 INFO:teuthology.orchestra.run.smithi005.stdout:115964117005 2022-02-21T17:13:32.946 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.8 2022-02-21T17:13:32.947 INFO:teuthology.orchestra.run.smithi005.stdout:115964117005 2022-02-21T17:13:32.948 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.3 2022-02-21T17:13:32.949 INFO:teuthology.orchestra.run.smithi005.stdout:137438953485 2022-02-21T17:13:32.950 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.7 2022-02-21T17:13:33.045 INFO:teuthology.orchestra.run.smithi005.stdout:146028888075 2022-02-21T17:13:33.046 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.10 2022-02-21T17:13:33.328 INFO:teuthology.orchestra.run.smithi005.stdout:137438953482 2022-02-21T17:13:33.483 INFO:teuthology.orchestra.run.smithi005.stdout:111669149709 2022-02-21T17:13:33.497 INFO:tasks.rook.ceph_manager.ceph:need seq 137438953484 got 137438953482 for osd.9 2022-02-21T17:13:33.603 INFO:teuthology.orchestra.run.smithi005.stdout:103079215117 2022-02-21T17:13:33.677 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149710 got 111669149709 for osd.0 2022-02-21T17:13:33.679 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215117 got 103079215117 for osd.5 2022-02-21T17:13:33.680 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:33.732 INFO:teuthology.orchestra.run.smithi005.stdout:73014444051 2022-02-21T17:13:33.760 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444050 got 73014444051 for osd.2 2022-02-21T17:13:33.760 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:33.778 INFO:teuthology.orchestra.run.smithi005.stdout:111669149709 2022-02-21T17:13:33.819 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149709 got 111669149709 for osd.4 2022-02-21T17:13:33.820 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:33.910 INFO:teuthology.orchestra.run.smithi005.stdout:128849018893 2022-02-21T17:13:33.936 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018893 got 128849018893 for osd.6 2022-02-21T17:13:33.937 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:33.977 INFO:teuthology.orchestra.run.smithi005.stdout:115964117005 2022-02-21T17:13:34.001 INFO:tasks.rook.ceph_manager.ceph:need seq 115964117005 got 115964117005 for osd.8 2022-02-21T17:13:34.001 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:34.044 INFO:teuthology.orchestra.run.smithi005.stdout:115964117005 2022-02-21T17:13:34.048 INFO:teuthology.orchestra.run.smithi005.stdout:146028888074 2022-02-21T17:13:34.056 INFO:teuthology.orchestra.run.smithi005.stdout:167503724550 2022-02-21T17:13:34.064 INFO:tasks.rook.ceph_manager.ceph:need seq 115964117005 got 115964117005 for osd.3 2022-02-21T17:13:34.065 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:34.076 INFO:tasks.rook.ceph_manager.ceph:need seq 146028888075 got 146028888074 for osd.10 2022-02-21T17:13:34.079 INFO:tasks.rook.ceph_manager.ceph:need seq 167503724550 got 167503724550 for osd.11 2022-02-21T17:13:34.079 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:34.115 INFO:teuthology.orchestra.run.smithi005.stdout:137438953485 2022-02-21T17:13:34.139 INFO:tasks.rook.ceph_manager.ceph:need seq 137438953485 got 137438953485 for osd.7 2022-02-21T17:13:34.139 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:34.155 INFO:teuthology.orchestra.run.smithi005.stdout:85899345936 2022-02-21T17:13:34.175 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345936 got 85899345936 for osd.1 2022-02-21T17:13:34.175 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:34.498 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.9 2022-02-21T17:13:34.678 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.0 2022-02-21T17:13:35.077 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.10 2022-02-21T17:13:35.087 INFO:teuthology.orchestra.run.smithi005.stdout:137438953484 2022-02-21T17:13:35.107 INFO:tasks.rook.ceph_manager.ceph:need seq 137438953484 got 137438953484 for osd.9 2022-02-21T17:13:35.108 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:35.194 INFO:teuthology.orchestra.run.smithi005.stdout:111669149710 2022-02-21T17:13:35.250 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149710 got 111669149710 for osd.0 2022-02-21T17:13:35.251 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:35.583 INFO:teuthology.orchestra.run.smithi005.stdout:146028888074 2022-02-21T17:13:35.604 INFO:tasks.rook.ceph_manager.ceph:need seq 146028888075 got 146028888074 for osd.10 2022-02-21T17:13:36.605 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph osd last-stat-seq osd.10 2022-02-21T17:13:37.131 INFO:teuthology.orchestra.run.smithi005.stdout:146028888076 2022-02-21T17:13:37.160 INFO:tasks.rook.ceph_manager.ceph:need seq 146028888075 got 146028888076 for osd.10 2022-02-21T17:13:37.161 DEBUG:teuthology.parallel:result is None 2022-02-21T17:13:37.161 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-02-21T17:13:37.162 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph pg dump --format=json 2022-02-21T17:13:37.660 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:37.662 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2022-02-21T17:13:37.683 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2022-02-21T17:13:36.933285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":934444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67660,"kb_used_data":6636,"kb_used_omap":0,"kb_used_meta":60928,"kb_avail":1124874164,"statfs":{"total":1151940427776,"available":1151871143936,"internally_reserved":0,"allocated":6795264,"data_stored":3954419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62390272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.043738"},"pg_stats":[{"pgid":"1.0","version":"36'137","reported_seq":255,"reported_epoch":43,"state":"active+clean","last_fresh":"2022-02-21T17:13:25.913906+0000","last_change":"2022-02-21T17:13:25.913906+0000","last_active":"2022-02-21T17:13:25.913906+0000","last_peered":"2022-02-21T17:13:25.913906+0000","last_clean":"2022-02-21T17:13:25.913906+0000","last_became_active":"2022-02-21T17:13:25.913331+0000","last_became_peered":"2022-02-21T17:13:25.913331+0000","last_unstale":"2022-02-21T17:13:25.913906+0000","last_undegraded":"2022-02-21T17:13:25.913906+0000","last_fullsized":"2022-02-21T17:13:25.913906+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-21T17:12:37.978117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-21T17:12:37.978117+0000","last_clean_scrub_stamp":"2022-02-21T17:12:37.978117+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-23T03:09:17.506647+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":934444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,10],"acting":[3,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":934444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2416640,"data_stored":2397816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":6}],"osd_stats":[{"osd":11,"up_from":39,"seq":167503724550,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5812,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739340,"statfs":{"total":95995035648,"available":95989084160,"internally_reserved":0,"allocated":962560,"data_stored":726084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.929}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1059999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.77}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4969999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]}]},{"osd":10,"up_from":34,"seq":146028888076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6132,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739020,"statfs":{"total":95995035648,"available":95988756480,"internally_reserved":0,"allocated":962560,"data_stored":726142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9570000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2519999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4870000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.04}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3359999999999999}]}]},{"osd":9,"up_from":32,"seq":137438953485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4159999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]}]},{"osd":7,"up_from":32,"seq":137438953486,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5612,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739540,"statfs":{"total":95995035648,"available":95989288960,"internally_reserved":0,"allocated":430080,"data_stored":195954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.51}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.915}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.976}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0760000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2050000000000001}]}]},{"osd":6,"up_from":30,"seq":128849018894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5360,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739792,"statfs":{"total":95995035648,"available":95989547008,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]}]},{"osd":8,"up_from":27,"seq":115964117006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5680,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739472,"statfs":{"total":95995035648,"available":95989219328,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.294}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3719999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5840000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.234}]}]},{"osd":1,"up_from":20,"seq":85899345937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5360,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739792,"statfs":{"total":95995035648,"available":95989547008,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8160000000000001}]},{"osd":2,"last update":"Mon Feb 21 17:13:29 2022","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.218,"5min":2.218,"15min":2.218},"last":2.2949999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":4.5469999999999997,"5min":4.5469999999999997,"15min":4.5469999999999997},"last":1.0980000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5329999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1600000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]}]},{"osd":2,"up_from":17,"seq":73014444051,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5424,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739728,"statfs":{"total":95995035648,"available":95989481472,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Mon Feb 21 17:13:31 2022","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.601}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8359999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9380000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4900000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.718}]}]},{"osd":5,"up_from":24,"seq":103079215118,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5744,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739408,"statfs":{"total":95995035648,"available":95989153792,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3000000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2680000000000007}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0660000000000007}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2080000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1769999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0850000000000009}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1449999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4410000000000007}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5329999999999999}]}]},{"osd":0,"up_from":26,"seq":111669149711,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5360,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739792,"statfs":{"total":95995035648,"available":95989547008,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]}]},{"osd":4,"up_from":26,"seq":111669149710,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5360,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739792,"statfs":{"total":95995035648,"available":95989547008,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.853}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7639999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7669999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2330000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.893}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0219999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":966656,"data_stored":726407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.921}]}]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-02-21T17:13:37.685 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph pg dump --format=json 2022-02-21T17:13:38.210 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:38.212 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2022-02-21T17:13:38.230 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2022-02-21T17:13:36.933285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":934444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67660,"kb_used_data":6636,"kb_used_omap":0,"kb_used_meta":60928,"kb_avail":1124874164,"statfs":{"total":1151940427776,"available":1151871143936,"internally_reserved":0,"allocated":6795264,"data_stored":3954419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62390272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.043738"},"pg_stats":[{"pgid":"1.0","version":"36'137","reported_seq":255,"reported_epoch":43,"state":"active+clean","last_fresh":"2022-02-21T17:13:25.913906+0000","last_change":"2022-02-21T17:13:25.913906+0000","last_active":"2022-02-21T17:13:25.913906+0000","last_peered":"2022-02-21T17:13:25.913906+0000","last_clean":"2022-02-21T17:13:25.913906+0000","last_became_active":"2022-02-21T17:13:25.913331+0000","last_became_peered":"2022-02-21T17:13:25.913331+0000","last_unstale":"2022-02-21T17:13:25.913906+0000","last_undegraded":"2022-02-21T17:13:25.913906+0000","last_fullsized":"2022-02-21T17:13:25.913906+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-21T17:12:37.978117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-21T17:12:37.978117+0000","last_clean_scrub_stamp":"2022-02-21T17:12:37.978117+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-23T03:09:17.506647+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":934444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,10],"acting":[3,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":934444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2416640,"data_stored":2397816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":6}],"osd_stats":[{"osd":11,"up_from":39,"seq":167503724550,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5812,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739340,"statfs":{"total":95995035648,"available":95989084160,"internally_reserved":0,"allocated":962560,"data_stored":726084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.929}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1059999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.77}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4969999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]}]},{"osd":10,"up_from":34,"seq":146028888076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6132,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739020,"statfs":{"total":95995035648,"available":95988756480,"internally_reserved":0,"allocated":962560,"data_stored":726142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9570000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2519999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4870000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.04}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3359999999999999}]}]},{"osd":9,"up_from":32,"seq":137438953485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5296,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739856,"statfs":{"total":95995035648,"available":95989612544,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4159999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]}]},{"osd":7,"up_from":32,"seq":137438953486,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5612,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739540,"statfs":{"total":95995035648,"available":95989288960,"internally_reserved":0,"allocated":430080,"data_stored":195954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.51}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.915}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.976}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0760000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2050000000000001}]}]},{"osd":6,"up_from":30,"seq":128849018894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5360,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739792,"statfs":{"total":95995035648,"available":95989547008,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]}]},{"osd":8,"up_from":27,"seq":115964117006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5680,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739472,"statfs":{"total":95995035648,"available":95989219328,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.294}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3719999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5840000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.234}]}]},{"osd":1,"up_from":20,"seq":85899345937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5360,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739792,"statfs":{"total":95995035648,"available":95989547008,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8160000000000001}]},{"osd":2,"last update":"Mon Feb 21 17:13:29 2022","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.218,"5min":2.218,"15min":2.218},"last":2.2949999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":4.5469999999999997,"5min":4.5469999999999997,"15min":4.5469999999999997},"last":1.0980000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5329999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1600000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.591}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]}]},{"osd":2,"up_from":17,"seq":73014444051,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5424,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739728,"statfs":{"total":95995035648,"available":95989481472,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Mon Feb 21 17:13:31 2022","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.601}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8359999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9380000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4900000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.718}]}]},{"osd":5,"up_from":24,"seq":103079215118,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5744,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739408,"statfs":{"total":95995035648,"available":95989153792,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3000000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2680000000000007}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0660000000000007}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2080000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1769999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0850000000000009}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1449999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.4410000000000007}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5329999999999999}]}]},{"osd":0,"up_from":26,"seq":111669149711,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5360,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739792,"statfs":{"total":95995035648,"available":95989547008,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]}]},{"osd":4,"up_from":26,"seq":111669149710,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5360,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739792,"statfs":{"total":95995035648,"available":95989547008,"internally_reserved":0,"allocated":434176,"data_stored":197479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.853}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7639999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7669999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2330000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.893}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0219999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":966656,"data_stored":726407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.921}]}]}],"pool_statfs":[{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-02-21T17:13:38.232 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-02-21T17:13:38.232 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-02-21T17:13:38.233 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-02-21T17:13:38.233 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph health --format=json 2022-02-21T17:13:38.833 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:38.833 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-02-21T17:13:38.852 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-02-21T17:13:38.853 INFO:tasks.rook:Rook complete, yielding 2022-02-21T17:13:38.853 INFO:teuthology.run_tasks:Running task rook.shell... 2022-02-21T17:13:38.870 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph -s 2022-02-21T17:13:39.486 INFO:teuthology.orchestra.run.smithi005.stdout: cluster: 2022-02-21T17:13:39.487 INFO:teuthology.orchestra.run.smithi005.stdout: id: 40f92d2b-27d2-4f26-ad3f-8c38961d94a2 2022-02-21T17:13:39.487 INFO:teuthology.orchestra.run.smithi005.stdout: health: HEALTH_OK 2022-02-21T17:13:39.487 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:39.487 INFO:teuthology.orchestra.run.smithi005.stdout: services: 2022-02-21T17:13:39.488 INFO:teuthology.orchestra.run.smithi005.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2022-02-21T17:13:39.488 INFO:teuthology.orchestra.run.smithi005.stdout: mgr: a(active, since 2m) 2022-02-21T17:13:39.488 INFO:teuthology.orchestra.run.smithi005.stdout: osd: 12 osds: 12 up (since 18s), 12 in (since 85s) 2022-02-21T17:13:39.488 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:39.489 INFO:teuthology.orchestra.run.smithi005.stdout: data: 2022-02-21T17:13:39.489 INFO:teuthology.orchestra.run.smithi005.stdout: pools: 1 pools, 1 pgs 2022-02-21T17:13:39.489 INFO:teuthology.orchestra.run.smithi005.stdout: objects: 2 objects, 577 KiB 2022-02-21T17:13:39.489 INFO:teuthology.orchestra.run.smithi005.stdout: usage: 66 MiB used, 1.0 TiB / 1.0 TiB avail 2022-02-21T17:13:39.489 INFO:teuthology.orchestra.run.smithi005.stdout: pgs: 1 active+clean 2022-02-21T17:13:39.490 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:13:39.513 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status 2022-02-21T17:13:40.230 INFO:teuthology.orchestra.run.smithi005.stdout:Backend: rook 2022-02-21T17:13:40.230 INFO:teuthology.orchestra.run.smithi005.stdout:Available: Yes 2022-02-21T17:13:40.265 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ps 2022-02-21T17:13:40.878 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-02-21T17:13:40.879 INFO:teuthology.orchestra.run.smithi005.stdout:crashcollector.crash smithi005 running (2m) 0s ago 2m - - f61e120f25ae 2022-02-21T17:13:40.879 INFO:teuthology.orchestra.run.smithi005.stdout:crashcollector.crash smithi033 running (88s) 0s ago 89s - - f61e120f25ae 2022-02-21T17:13:40.879 INFO:teuthology.orchestra.run.smithi005.stdout:crashcollector.crash smithi067 running (2m) 0s ago 2m - - f61e120f25ae 2022-02-21T17:13:40.880 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.a smithi033 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:13:40.880 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi067 running (5m) 0s ago 5m - - f61e120f25ae 2022-02-21T17:13:40.880 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi033 running (5m) 0s ago 5m - - f61e120f25ae 2022-02-21T17:13:40.880 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:13:40.881 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (106s) 0s ago 107s - - f61e120f25ae 2022-02-21T17:13:40.881 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi067 running (105s) 0s ago 105s - - f61e120f25ae 2022-02-21T17:13:40.881 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10 smithi005 running (83s) 0s ago 83s - - f61e120f25ae 2022-02-21T17:13:40.882 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11 smithi033 running (38s) 0s ago 39s - - f61e120f25ae 2022-02-21T17:13:40.882 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi067 running (105s) 0s ago 106s - - f61e120f25ae 2022-02-21T17:13:40.882 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (101s) 0s ago 102s - - f61e120f25ae 2022-02-21T17:13:40.882 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi005 running (99s) 0s ago 100s - - f61e120f25ae 2022-02-21T17:13:40.883 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi067 running (91s) 0s ago 91s - - f61e120f25ae 2022-02-21T17:13:40.883 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi033 running (89s) 0s ago 89s - - f61e120f25ae 2022-02-21T17:13:40.883 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi033 running (88s) 0s ago 88s - - f61e120f25ae 2022-02-21T17:13:40.883 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8 smithi067 running (86s) 0s ago 86s - - f61e120f25ae 2022-02-21T17:13:40.884 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9 smithi033 running (84s) 0s ago 85s - - f61e120f25ae 2022-02-21T17:13:40.896 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls 2022-02-21T17:13:41.594 INFO:teuthology.orchestra.run.smithi005.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-02-21T17:13:41.594 INFO:teuthology.orchestra.run.smithi005.stdout:crash 3/3 0s ago 2m * 2022-02-21T17:13:41.594 INFO:teuthology.orchestra.run.smithi005.stdout:mgr 1/1 0s ago 3m count:1 2022-02-21T17:13:41.595 INFO:teuthology.orchestra.run.smithi005.stdout:mon 3/3 0s ago 5m count:3 2022-02-21T17:13:41.595 INFO:teuthology.orchestra.run.smithi005.stdout:osd 24 0s ago 108s 2022-02-21T17:13:41.595 INFO:teuthology.orchestra.run.smithi005.stdout:osd.all-available-devices 0 0s ago - * 2022-02-21T17:13:41.618 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch device ls 2022-02-21T17:13:42.142 INFO:teuthology.orchestra.run.smithi005.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2022-02-21T17:13:42.143 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T17:13:42.143 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T17:13:42.143 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T17:13:42.143 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T17:13:42.144 INFO:teuthology.orchestra.run.smithi005.stdout:smithi033 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T17:13:42.144 INFO:teuthology.orchestra.run.smithi005.stdout:smithi033 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T17:13:42.144 INFO:teuthology.orchestra.run.smithi005.stdout:smithi033 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T17:13:42.144 INFO:teuthology.orchestra.run.smithi005.stdout:smithi033 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T17:13:42.144 INFO:teuthology.orchestra.run.smithi005.stdout:smithi067 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T17:13:42.145 INFO:teuthology.orchestra.run.smithi005.stdout:smithi067 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T17:13:42.145 INFO:teuthology.orchestra.run.smithi005.stdout:smithi067 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T17:13:42.145 INFO:teuthology.orchestra.run.smithi005.stdout:smithi067 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T17:13:42.172 INFO:teuthology.run_tasks:Running task exec... 2022-02-21T17:13:42.185 INFO:teuthology.task.exec:Executing custom commands... 2022-02-21T17:13:42.186 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi005.front.sepia.ceph.com 2022-02-21T17:13:42.186 DEBUG:teuthology.orchestra.run.smithi005:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-02-21T17:13:42.186 DEBUG:teuthology.orchestra.run.smithi005:> toolbox() { 2022-02-21T17:13:42.187 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-02-21T17:13:42.187 DEBUG:teuthology.orchestra.run.smithi005:> } 2022-02-21T17:13:42.187 DEBUG:teuthology.orchestra.run.smithi005:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-02-21T17:13:42.188 DEBUG:teuthology.orchestra.run.smithi005:> toolbox ceph orch osd rm 0 --force 2022-02-21T17:13:42.188 DEBUG:teuthology.orchestra.run.smithi005:> removed_pv="" 2022-02-21T17:13:42.188 DEBUG:teuthology.orchestra.run.smithi005:> while [ "$removed_pv" = "" ] 2022-02-21T17:13:42.189 DEBUG:teuthology.orchestra.run.smithi005:> do 2022-02-21T17:13:42.189 DEBUG:teuthology.orchestra.run.smithi005:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-02-21T17:13:42.189 DEBUG:teuthology.orchestra.run.smithi005:> sleep 3s 2022-02-21T17:13:42.189 DEBUG:teuthology.orchestra.run.smithi005:> done 2022-02-21T17:13:42.190 DEBUG:teuthology.orchestra.run.smithi005:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-02-21T17:13:42.190 DEBUG:teuthology.orchestra.run.smithi005:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-02-21T17:13:42.190 DEBUG:teuthology.orchestra.run.smithi005:> toolbox ceph orch device zap $host $target_path --force 2022-02-21T17:13:42.190 DEBUG:teuthology.orchestra.run.smithi005:> zap_completion="0" 2022-02-21T17:13:42.191 DEBUG:teuthology.orchestra.run.smithi005:> while [ "$zap_completion" = "0" ] 2022-02-21T17:13:42.191 DEBUG:teuthology.orchestra.run.smithi005:> do 2022-02-21T17:13:42.191 DEBUG:teuthology.orchestra.run.smithi005:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-02-21T17:13:42.192 DEBUG:teuthology.orchestra.run.smithi005:> sleep 3s 2022-02-21T17:13:42.192 DEBUG:teuthology.orchestra.run.smithi005:> done 2022-02-21T17:13:42.192 DEBUG:teuthology.orchestra.run.smithi005:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-02-21T17:13:42.192 DEBUG:teuthology.orchestra.run.smithi005:> toolbox ceph orch apply osd --all-available-devices 2022-02-21T17:13:42.193 DEBUG:teuthology.orchestra.run.smithi005:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-02-21T17:13:42.193 DEBUG:teuthology.orchestra.run.smithi005:> num_osd="0" 2022-02-21T17:13:42.193 DEBUG:teuthology.orchestra.run.smithi005:> while [ "$num_osd" != "$orig_num_osd" ] 2022-02-21T17:13:42.194 DEBUG:teuthology.orchestra.run.smithi005:> do 2022-02-21T17:13:42.194 DEBUG:teuthology.orchestra.run.smithi005:> echo "waiting for osd to come back up" 2022-02-21T17:13:42.194 DEBUG:teuthology.orchestra.run.smithi005:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-02-21T17:13:42.194 DEBUG:teuthology.orchestra.run.smithi005:> sleep 30s 2022-02-21T17:13:42.195 DEBUG:teuthology.orchestra.run.smithi005:> done 2022-02-21T17:13:42.195 DEBUG:teuthology.orchestra.run.smithi005:> ' 2022-02-21T17:13:42.214 INFO:teuthology.orchestra.run.smithi005.stderr:++ toolbox ceph osd stat 2022-02-21T17:13:42.214 INFO:teuthology.orchestra.run.smithi005.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T17:13:42.215 INFO:teuthology.orchestra.run.smithi005.stderr:++ cut -f3 -d ' ' 2022-02-21T17:13:44.655 INFO:teuthology.orchestra.run.smithi005.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T17:13:45.194 INFO:teuthology.orchestra.run.smithi005.stderr:+ orig_num_osd=12 2022-02-21T17:13:45.195 INFO:teuthology.orchestra.run.smithi005.stderr:+ toolbox ceph orch osd rm 0 --force 2022-02-21T17:13:45.195 INFO:teuthology.orchestra.run.smithi005.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-02-21T17:13:45.269 INFO:teuthology.orchestra.run.smithi005.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T17:13:45.907 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:45.898596 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-02-21T17:13:45.907 INFO:tasks.rook.operator.smithi005.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", AllowUnsupported: true}, 2022-02-21T17:13:45.907 INFO:tasks.rook.operator.smithi005.stdout: Storage: v1.StorageScopeSpec{ 2022-02-21T17:13:45.908 INFO:tasks.rook.operator.smithi005.stdout: ... // 3 identical fields 2022-02-21T17:13:45.908 INFO:tasks.rook.operator.smithi005.stdout: Config: nil, 2022-02-21T17:13:45.909 INFO:tasks.rook.operator.smithi005.stdout: Selection: {}, 2022-02-21T17:13:45.909 INFO:tasks.rook.operator.smithi005.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-02-21T17:13:45.909 INFO:tasks.rook.operator.smithi005.stdout:- { 2022-02-21T17:13:45.909 INFO:tasks.rook.operator.smithi005.stdout:- Name: "smithi005-lv-1", 2022-02-21T17:13:45.910 INFO:tasks.rook.operator.smithi005.stdout:- Count: 1, 2022-02-21T17:13:45.910 INFO:tasks.rook.operator.smithi005.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-02-21T17:13:45.910 INFO:tasks.rook.operator.smithi005.stdout:- }, 2022-02-21T17:13:45.911 INFO:tasks.rook.operator.smithi005.stdout: {Name: "smithi005-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi005-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-02-21T17:13:45.911 INFO:tasks.rook.operator.smithi005.stdout: {Name: "smithi005-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi005-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-02-21T17:13:45.911 INFO:tasks.rook.operator.smithi005.stdout: ... // 9 identical elements 2022-02-21T17:13:45.911 INFO:tasks.rook.operator.smithi005.stdout: }, 2022-02-21T17:13:45.912 INFO:tasks.rook.operator.smithi005.stdout: }, 2022-02-21T17:13:45.912 INFO:tasks.rook.operator.smithi005.stdout: Annotations: nil, 2022-02-21T17:13:45.912 INFO:tasks.rook.operator.smithi005.stdout: Labels: nil, 2022-02-21T17:13:45.912 INFO:tasks.rook.operator.smithi005.stdout: ... // 20 identical fields 2022-02-21T17:13:45.912 INFO:tasks.rook.operator.smithi005.stdout: } 2022-02-21T17:13:45.913 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:45.898733 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-02-21T17:13:45.913 INFO:tasks.rook.operator.smithi005.stdout:I0221 17:13:45.898929 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-02-21T17:13:45.913 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:45.899367 I | operator: successfully started the controller-runtime manager 2022-02-21T17:13:45.921 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:45.914948 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T17:13:45.921 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:45.914966 I | operator: watching all namespaces for Ceph CRs 2022-02-21T17:13:45.921 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:45.914996 I | operator: setting up schemes 2022-02-21T17:13:45.932 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:45.918464 I | operator: setting up the controller-runtime manager 2022-02-21T17:13:46.128 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.122163 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T17:13:46.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176473 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T17:13:46.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176521 I | ceph-cluster-controller: successfully started 2022-02-21T17:13:46.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176605 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T17:13:46.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176642 I | ceph-crashcollector-controller: successfully started 2022-02-21T17:13:46.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176688 I | ceph-block-pool-controller: successfully started 2022-02-21T17:13:46.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176738 I | ceph-object-store-user-controller: successfully started 2022-02-21T17:13:46.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176770 I | ceph-object-realm-controller: successfully started 2022-02-21T17:13:46.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176787 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T17:13:46.186 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176800 I | ceph-object-zone-controller: successfully started 2022-02-21T17:13:46.186 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.176987 I | ceph-object-controller: successfully started 2022-02-21T17:13:46.186 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177032 I | ceph-file-controller: successfully started 2022-02-21T17:13:46.186 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177066 I | ceph-nfs-controller: successfully started 2022-02-21T17:13:46.186 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177101 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T17:13:46.187 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177129 I | ceph-client-controller: successfully started 2022-02-21T17:13:46.187 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177159 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T17:13:46.187 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177209 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T17:13:46.187 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177230 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T17:13:46.188 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177251 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T17:13:46.188 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177275 I | ceph-bucket-topic: successfully started 2022-02-21T17:13:46.188 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177302 I | ceph-bucket-notification: successfully started 2022-02-21T17:13:46.188 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177325 I | ceph-bucket-notification: successfully started 2022-02-21T17:13:46.189 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.177340 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T17:13:46.190 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.179031 I | operator: starting the controller-runtime manager 2022-02-21T17:13:46.287 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.280704 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T17:13:46.287 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.280755 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T17:13:46.287 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.280764 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T17:13:46.292 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.286448 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T17:13:46.433 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.426895 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T17:13:46.517 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.509663 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:13:46.517 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.509734 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:13:46.518 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.509784 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-02-21T17:13:46.518 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.509817 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-02-21T17:13:46.521 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.512273 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-02-21T17:13:46.521 INFO:tasks.rook.operator.smithi005.stdout:I0221 17:13:46.512361 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-02-21T17:13:46.835 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.828946 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-02-21T17:13:46.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832645 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-02-21T17:13:46.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832672 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-02-21T17:13:46.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832682 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-02-21T17:13:46.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832689 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-02-21T17:13:46.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832702 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-02-21T17:13:46.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832714 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-02-21T17:13:46.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832724 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-02-21T17:13:46.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832733 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-02-21T17:13:46.843 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832742 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-02-21T17:13:46.843 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832752 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-02-21T17:13:46.843 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832763 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-02-21T17:13:46.844 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832779 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-02-21T17:13:46.844 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832794 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-02-21T17:13:46.844 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832806 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-02-21T17:13:46.845 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832818 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-02-21T17:13:46.845 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832833 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-02-21T17:13:46.845 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832937 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:13:46.846 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:46.832956 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:13:47.195 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:47.181734 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T17:13:50.018 INFO:teuthology.orchestra.run.smithi005.stdout:Success 2022-02-21T17:13:50.050 INFO:teuthology.orchestra.run.smithi005.stderr:+ removed_pv= 2022-02-21T17:13:50.050 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' '' = '' ']' 2022-02-21T17:13:50.051 INFO:teuthology.orchestra.run.smithi005.stderr:++ kubectl get pv 2022-02-21T17:13:50.051 INFO:teuthology.orchestra.run.smithi005.stderr:++ grep Released 2022-02-21T17:13:50.052 INFO:teuthology.orchestra.run.smithi005.stderr:++ cut -f1 -d ' ' 2022-02-21T17:13:50.110 INFO:teuthology.orchestra.run.smithi005.stderr:+ removed_pv= 2022-02-21T17:13:50.110 INFO:teuthology.orchestra.run.smithi005.stderr:+ sleep 3s 2022-02-21T17:13:51.975 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:51.966523 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-02-21T17:13:52.827 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820082 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-02-21T17:13:52.833 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820116 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-02-21T17:13:52.834 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820126 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T17:13:52.834 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820132 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T17:13:52.834 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820139 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T17:13:52.835 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820145 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T17:13:52.835 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820151 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T17:13:52.836 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820156 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T17:13:52.836 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820163 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T17:13:52.837 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820169 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T17:13:52.837 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820175 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T17:13:52.837 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820181 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T17:13:52.838 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820187 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-02-21T17:13:52.838 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820193 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-02-21T17:13:52.839 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820199 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-02-21T17:13:52.839 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820205 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-02-21T17:13:52.839 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820211 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-02-21T17:13:52.840 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820216 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-02-21T17:13:52.840 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820223 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-02-21T17:13:52.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820229 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T17:13:52.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820236 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T17:13:52.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820242 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-02-21T17:13:52.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820248 I | ceph-csi: Kubernetes version is 1.23 2022-02-21T17:13:52.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820256 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-02-21T17:13:52.843 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.820262 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-02-21T17:13:52.853 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.845734 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-02-21T17:13:52.861 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.854626 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:13:52.862 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.854654 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:13:52.862 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.854662 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:13:52.862 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.854670 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:13:52.862 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.854680 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:13:52.863 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.854689 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:13:52.863 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:52.854699 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-02-21T17:13:53.019 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:53.011644 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:13:53.019 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:53.011675 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:13:53.019 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:53.011687 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-02-21T17:13:53.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:53.055774 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-02-21T17:13:53.109 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' '' = '' ']' 2022-02-21T17:13:53.110 INFO:teuthology.orchestra.run.smithi005.stderr:++ kubectl get pv 2022-02-21T17:13:53.111 INFO:teuthology.orchestra.run.smithi005.stderr:++ grep Released 2022-02-21T17:13:53.111 INFO:teuthology.orchestra.run.smithi005.stderr:++ cut -f1 -d ' ' 2022-02-21T17:13:53.475 INFO:teuthology.orchestra.run.smithi005.stderr:+ removed_pv=smithi005-lv-1 2022-02-21T17:13:53.476 INFO:teuthology.orchestra.run.smithi005.stderr:+ sleep 3s 2022-02-21T17:13:54.397 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.389337 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:13:54.397 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.389370 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:13:54.397 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.389380 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-02-21T17:13:54.441 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.433900 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:13:54.441 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.433923 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:13:54.441 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.433928 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-02-21T17:13:54.488 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.480469 I | ceph-csi: successfully started CSI CephFS driver 2022-02-21T17:13:54.675 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.665827 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-02-21T17:13:54.678 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.670759 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-02-21T17:13:54.678 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.670789 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-02-21T17:13:54.685 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:54.675593 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-02-21T17:13:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' smithi005-lv-1 = '' ']' 2022-02-21T17:13:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:++ kubectl get pv smithi005-lv-1 -o 'jsonpath={.spec.local.path}' 2022-02-21T17:13:56.540 INFO:teuthology.orchestra.run.smithi005.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-02-21T17:13:56.540 INFO:teuthology.orchestra.run.smithi005.stderr:++ echo smithi005-lv-1 2022-02-21T17:13:56.540 INFO:teuthology.orchestra.run.smithi005.stderr:++ cut -f1 -d - 2022-02-21T17:13:56.541 INFO:teuthology.orchestra.run.smithi005.stderr:+ host=smithi005 2022-02-21T17:13:56.542 INFO:teuthology.orchestra.run.smithi005.stderr:+ toolbox ceph orch device zap smithi005 /dev/vg_nvme/lv_1 --force 2022-02-21T17:13:56.542 INFO:teuthology.orchestra.run.smithi005.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi005 /dev/vg_nvme/lv_1 --force 2022-02-21T17:13:56.699 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:56.691447 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T17:13:56.700 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:56.691476 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T17:13:56.700 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:56.691491 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T17:13:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T17:13:57.071 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:57.063919 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:13:57.261 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:57.253459 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:13:57.265 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:57.253778 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:13:57.561 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_1 on smithi005 zapped 2022-02-21T17:13:57.584 INFO:teuthology.orchestra.run.smithi005.stderr:+ zap_completion=0 2022-02-21T17:13:57.584 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' 0 = 0 ']' 2022-02-21T17:13:57.584 INFO:teuthology.orchestra.run.smithi005.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-02-21T17:13:57.648 INFO:teuthology.orchestra.run.smithi005.stderr:+ zap_completion= 2022-02-21T17:13:57.649 INFO:teuthology.orchestra.run.smithi005.stderr:+ sleep 3s 2022-02-21T17:13:57.687 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:57.679257 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T17:13:57.810 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:57.802681 I | op-mon: start running mons 2022-02-21T17:13:57.936 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:57.928297 I | op-mon: parsing mon endpoints: a=10.110.173.53:6789,b=10.106.73.178:6789,c=10.109.91.196:6789 2022-02-21T17:13:57.983 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:57.972616 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.173.53:6789","10.106.73.178:6789","10.109.91.196:6789"]}] data:b=10.106.73.178:6789,c=10.109.91.196:6789,a=10.110.173.53:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:2] 2022-02-21T17:13:58.230 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:58.222117 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:13:58.230 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:58.222443 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:13:59.591 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:59.583221 I | op-mon: targeting the mon count 3 2022-02-21T17:13:59.598 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:59.590232 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:13:59.973 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:59.964864 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:13:59.973 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:13:59.964896 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:14:00.310 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:00.302101 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:14:00.310 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:00.302137 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:14:00.651 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' '' = 0 ']' 2022-02-21T17:14:00.651 INFO:teuthology.orchestra.run.smithi005.stderr:+ kubectl patch pv smithi005-lv-1 -p '{"spec":{"claimRef": null}}' 2022-02-21T17:14:00.661 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:00.653065 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:14:00.661 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:00.653099 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:14:00.767 INFO:teuthology.orchestra.run.smithi005.stdout:persistentvolume/smithi005-lv-1 patched 2022-02-21T17:14:00.769 INFO:teuthology.orchestra.run.smithi005.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-02-21T17:14:00.769 INFO:teuthology.orchestra.run.smithi005.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-02-21T17:14:00.992 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:00.984433 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:14:00.997 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:00.984475 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T17:14:01.007 INFO:teuthology.orchestra.run.smithi005.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T17:14:01.352 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.343013 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T17:14:01.353 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.343042 I | op-mon: checking for basic quorum with existing mons 2022-02-21T17:14:01.445 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.436764 I | op-mon: mon "a" endpoint is [v2:10.110.173.53:3300,v1:10.110.173.53:6789] 2022-02-21T17:14:01.620 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.612565 I | op-mon: mon "b" endpoint is [v2:10.106.73.178:3300,v1:10.106.73.178:6789] 2022-02-21T17:14:01.756 INFO:teuthology.orchestra.run.smithi005.stdout:['Success'] 2022-02-21T17:14:01.764 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.752857 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-02-21T17:14:01.765 INFO:tasks.rook.operator.smithi005.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", AllowUnsupported: true}, 2022-02-21T17:14:01.766 INFO:tasks.rook.operator.smithi005.stdout: Storage: v1.StorageScopeSpec{ 2022-02-21T17:14:01.766 INFO:tasks.rook.operator.smithi005.stdout: ... // 3 identical fields 2022-02-21T17:14:01.767 INFO:tasks.rook.operator.smithi005.stdout: Config: nil, 2022-02-21T17:14:01.767 INFO:tasks.rook.operator.smithi005.stdout: Selection: {}, 2022-02-21T17:14:01.767 INFO:tasks.rook.operator.smithi005.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-02-21T17:14:01.768 INFO:tasks.rook.operator.smithi005.stdout: ... // 9 identical elements 2022-02-21T17:14:01.768 INFO:tasks.rook.operator.smithi005.stdout: {Name: "smithi067-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi067-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-02-21T17:14:01.769 INFO:tasks.rook.operator.smithi005.stdout: {Name: "smithi067-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi067-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-02-21T17:14:01.769 INFO:tasks.rook.operator.smithi005.stdout:+ { 2022-02-21T17:14:01.770 INFO:tasks.rook.operator.smithi005.stdout:+ Name: "smithi005-lv-1", 2022-02-21T17:14:01.770 INFO:tasks.rook.operator.smithi005.stdout:+ Count: 1, 2022-02-21T17:14:01.770 INFO:tasks.rook.operator.smithi005.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-02-21T17:14:01.771 INFO:tasks.rook.operator.smithi005.stdout:+ }, 2022-02-21T17:14:01.771 INFO:tasks.rook.operator.smithi005.stdout: }, 2022-02-21T17:14:01.772 INFO:tasks.rook.operator.smithi005.stdout: }, 2022-02-21T17:14:01.772 INFO:tasks.rook.operator.smithi005.stdout: Annotations: nil, 2022-02-21T17:14:01.773 INFO:tasks.rook.operator.smithi005.stdout: Labels: nil, 2022-02-21T17:14:01.773 INFO:tasks.rook.operator.smithi005.stdout: ... // 20 identical fields 2022-02-21T17:14:01.773 INFO:tasks.rook.operator.smithi005.stdout: } 2022-02-21T17:14:01.774 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.753013 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-02-21T17:14:01.774 INFO:tasks.rook.operator.smithi005.stdout:I0221 17:14:01.753307 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-02-21T17:14:01.776 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.766856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: failed to create mon service: failed to create service for mon c: failed to update service rook-ceph-mon-c. could not get existing service rook-ceph-mon-c in order to update. context canceled 2022-02-21T17:14:01.776 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.766997 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T17:14:01.777 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.767103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to load cluster info: failed to get mon secrets: context canceled 2022-02-21T17:14:01.777 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.767170 I | operator: successfully started the controller-runtime manager 2022-02-21T17:14:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-02-21T17:14:01.796 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.785088 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T17:14:01.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.785113 I | operator: watching all namespaces for Ceph CRs 2022-02-21T17:14:01.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.785139 I | operator: setting up schemes 2022-02-21T17:14:01.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.788238 I | operator: setting up the controller-runtime manager 2022-02-21T17:14:01.839 INFO:teuthology.orchestra.run.smithi005.stdout:job.batch "rook-ceph-device-zap" deleted 2022-02-21T17:14:01.845 INFO:teuthology.orchestra.run.smithi005.stderr:+ num_osd=0 2022-02-21T17:14:01.845 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' 0 '!=' 12 ']' 2022-02-21T17:14:01.845 INFO:teuthology.orchestra.run.smithi005.stderr:+ echo 'waiting for osd to come back up' 2022-02-21T17:14:01.846 INFO:teuthology.orchestra.run.smithi005.stdout:waiting for osd to come back up 2022-02-21T17:14:01.846 INFO:teuthology.orchestra.run.smithi005.stderr:++ toolbox ceph osd stat 2022-02-21T17:14:01.847 INFO:teuthology.orchestra.run.smithi005.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T17:14:01.847 INFO:teuthology.orchestra.run.smithi005.stderr:++ cut -f3 -d ' ' 2022-02-21T17:14:02.001 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:01.990912 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T17:14:02.097 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.088807 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T17:14:02.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.088854 I | ceph-cluster-controller: successfully started 2022-02-21T17:14:02.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.088927 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T17:14:02.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.088957 I | ceph-crashcollector-controller: successfully started 2022-02-21T17:14:02.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089001 I | ceph-block-pool-controller: successfully started 2022-02-21T17:14:02.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089029 I | ceph-object-store-user-controller: successfully started 2022-02-21T17:14:02.099 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089055 I | ceph-object-realm-controller: successfully started 2022-02-21T17:14:02.099 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089079 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T17:14:02.099 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089104 I | ceph-object-zone-controller: successfully started 2022-02-21T17:14:02.103 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089310 I | ceph-object-controller: successfully started 2022-02-21T17:14:02.104 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089360 I | ceph-file-controller: successfully started 2022-02-21T17:14:02.104 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089409 I | ceph-nfs-controller: successfully started 2022-02-21T17:14:02.104 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089441 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T17:14:02.104 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089472 I | ceph-client-controller: successfully started 2022-02-21T17:14:02.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089492 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T17:14:02.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089544 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T17:14:02.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089567 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T17:14:02.106 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089603 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T17:14:02.106 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089637 I | ceph-bucket-topic: successfully started 2022-02-21T17:14:02.107 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089654 I | ceph-bucket-notification: successfully started 2022-02-21T17:14:02.107 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089669 I | ceph-bucket-notification: successfully started 2022-02-21T17:14:02.107 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.089684 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T17:14:02.107 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.090839 I | operator: starting the controller-runtime manager 2022-02-21T17:14:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T17:14:02.204 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.195005 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T17:14:02.207 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.195043 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T17:14:02.207 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.195055 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T17:14:02.208 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.197997 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T17:14:02.438 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.430036 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T17:14:02.613 INFO:teuthology.orchestra.run.smithi005.stderr:+ num_osd=11 2022-02-21T17:14:02.613 INFO:teuthology.orchestra.run.smithi005.stderr:+ sleep 30s 2022-02-21T17:14:02.791 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.782477 I | op-mon: parsing mon endpoints: b=10.106.73.178:6789,c=10.109.91.196:6789,a=10.110.173.53:6789 2022-02-21T17:14:02.791 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.782572 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-02-21T17:14:02.791 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.782588 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-02-21T17:14:02.795 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:02.784852 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-02-21T17:14:02.795 INFO:tasks.rook.operator.smithi005.stdout:I0221 17:14:02.784943 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-02-21T17:14:03.045 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.037266 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-02-21T17:14:03.051 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040502 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-02-21T17:14:03.052 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040542 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-02-21T17:14:03.052 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040556 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-02-21T17:14:03.052 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040569 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-02-21T17:14:03.052 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040581 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-02-21T17:14:03.053 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040593 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-02-21T17:14:03.053 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040608 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-02-21T17:14:03.053 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040623 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-02-21T17:14:03.053 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040639 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-02-21T17:14:03.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040653 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-02-21T17:14:03.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040667 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-02-21T17:14:03.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040682 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-02-21T17:14:03.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040695 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-02-21T17:14:03.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040706 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-02-21T17:14:03.055 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040720 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-02-21T17:14:03.055 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040739 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-02-21T17:14:03.055 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040836 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:14:03.055 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.040858 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:14:03.244 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.235873 I | op-mon: parsing mon endpoints: b=10.106.73.178:6789,c=10.109.91.196:6789,a=10.110.173.53:6789 2022-02-21T17:14:03.851 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:03.842606 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T17:14:12.021 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010127 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-02-21T17:14:12.021 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010153 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-02-21T17:14:12.022 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010160 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T17:14:12.022 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010164 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-02-21T17:14:12.023 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010169 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T17:14:12.023 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010172 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-02-21T17:14:12.023 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010176 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T17:14:12.023 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010180 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-02-21T17:14:12.024 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010184 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T17:14:12.024 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010188 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-02-21T17:14:12.024 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010192 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T17:14:12.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010196 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-02-21T17:14:12.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010200 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-02-21T17:14:12.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010203 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-02-21T17:14:12.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010207 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-02-21T17:14:12.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010211 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-02-21T17:14:12.026 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010215 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-02-21T17:14:12.026 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010219 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-02-21T17:14:12.026 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010223 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-02-21T17:14:12.027 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010228 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T17:14:12.027 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010232 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-02-21T17:14:12.027 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010236 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-02-21T17:14:12.027 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010240 I | ceph-csi: Kubernetes version is 1.23 2022-02-21T17:14:12.028 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010245 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-02-21T17:14:12.028 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.010251 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-02-21T17:14:12.056 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.047346 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-02-21T17:14:12.062 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.052622 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:14:12.062 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.052640 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:14:12.062 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.052644 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:14:12.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.052646 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:14:12.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.052649 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:14:12.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.052651 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:14:12.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.052654 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-02-21T17:14:12.137 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.126481 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T17:14:12.137 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.126513 I | ceph-cluster-controller: validating ceph version from provided image 2022-02-21T17:14:12.138 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.126533 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10781 quincy", pursuing 2022-02-21T17:14:12.141 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.131032 I | op-mon: parsing mon endpoints: b=10.106.73.178:6789,c=10.109.91.196:6789,a=10.110.173.53:6789 2022-02-21T17:14:12.191 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.180846 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:14:12.191 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.180864 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:14:12.191 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.180868 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-02-21T17:14:12.216 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.207020 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:14:12.229 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.218826 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-02-21T17:14:12.242 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.233677 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:14:12.380 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.370967 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-02-21T17:14:12.381 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.370991 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-02-21T17:14:12.381 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.370996 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-02-21T17:14:12.448 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.439102 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-02-21T17:14:12.448 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.439120 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-02-21T17:14:12.449 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.439124 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-02-21T17:14:12.487 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.477937 I | ceph-csi: successfully started CSI CephFS driver 2022-02-21T17:14:12.681 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.672060 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10781 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e" 2022-02-21T17:14:12.820 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.811748 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-02-21T17:14:12.826 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.817084 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-02-21T17:14:12.826 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.817116 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-02-21T17:14:12.832 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:12.821677 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-02-21T17:14:13.041 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:13.032255 I | op-mon: start running mons 2022-02-21T17:14:13.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:13.383683 I | op-mon: parsing mon endpoints: b=10.106.73.178:6789,c=10.109.91.196:6789,a=10.110.173.53:6789 2022-02-21T17:14:14.394 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:14.384853 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.73.178:6789","10.109.91.196:6789","10.110.173.53:6789"]}] data:c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:2] 2022-02-21T17:14:14.993 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:14.984045 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:14:14.997 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:14.984463 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:14:16.599 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:16.586970 I | op-mon: targeting the mon count 3 2022-02-21T17:14:16.606 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:16.596361 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:14:16.968 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:16.958522 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-02-21T17:14:17.050 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:16.958555 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:14:17.322 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:17.312525 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-02-21T17:14:17.322 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:17.312567 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:14:17.661 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:17.651388 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-02-21T17:14:17.661 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:17.651427 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:14:18.007 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:17.997194 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-02-21T17:14:18.007 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:17.997228 I | op-config: deleting "log file" option from the mon configuration database 2022-02-21T17:14:18.345 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:18.335917 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-02-21T17:14:18.345 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:18.335946 I | op-mon: checking for basic quorum with existing mons 2022-02-21T17:14:18.430 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:18.421005 I | op-mon: mon "b" endpoint is [v2:10.106.73.178:3300,v1:10.106.73.178:6789] 2022-02-21T17:14:18.584 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:18.574447 I | op-mon: mon "c" endpoint is [v2:10.109.91.196:3300,v1:10.109.91.196:6789] 2022-02-21T17:14:18.995 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:18.985500 I | op-mon: mon "a" endpoint is [v2:10.110.173.53:3300,v1:10.110.173.53:6789] 2022-02-21T17:14:19.595 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:19.585486 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.73.178:6789","10.109.91.196:6789","10.110.173.53:6789"]}] data:c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 mapping:{"node":{"a":{"Name":"smithi067","Hostname":"smithi067","Address":"172.21.15.67"},"b":{"Name":"smithi033","Hostname":"smithi033","Address":"172.21.15.33"},"c":{"Name":"smithi005","Hostname":"smithi005","Address":"172.21.15.5"}}} maxMonId:2] 2022-02-21T17:14:20.194 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:20.184928 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-02-21T17:14:20.198 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:20.185229 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-02-21T17:14:20.618 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:20.607726 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-02-21T17:14:20.632 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:20.622552 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-02-21T17:14:20.632 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:20.622585 I | op-mon: waiting for mon quorum with [b c a] 2022-02-21T17:14:21.204 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:21.194960 I | op-mon: mons running: [b c a] 2022-02-21T17:14:21.630 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:21.620855 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:14:21.635 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:21.626397 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-02-21T17:14:21.645 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:21.636285 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-02-21T17:14:21.646 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:21.636305 I | op-mon: waiting for mon quorum with [b c a] 2022-02-21T17:14:21.806 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:21.796056 I | op-mon: mons running: [b c a] 2022-02-21T17:14:22.217 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:22.207437 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:14:22.225 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:22.215607 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-02-21T17:14:22.235 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:22.224447 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-02-21T17:14:22.235 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:22.224474 I | op-mon: waiting for mon quorum with [b c a] 2022-02-21T17:14:22.416 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:22.406916 I | op-mon: mons running: [b c a] 2022-02-21T17:14:22.836 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:22.825952 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:14:22.836 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:22.825983 I | op-mon: mons created: 3 2022-02-21T17:14:23.260 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:23.249983 I | op-mon: waiting for mon quorum with [b c a] 2022-02-21T17:14:23.301 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:23.285715 I | op-mon: mons running: [b c a] 2022-02-21T17:14:23.741 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:23.731226 I | op-mon: Monitors in quorum: [a b c] 2022-02-21T17:14:23.741 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:23.731272 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-02-21T17:14:24.167 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:24.156136 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-02-21T17:14:24.594 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:24.584428 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-02-21T17:14:25.078 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:25.068089 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-02-21T17:14:25.535 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:25.525075 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-02-21T17:14:25.535 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:25.525106 I | cephclient: getting or creating ceph auth key "client.crash" 2022-02-21T17:14:26.012 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:26.002207 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-02-21T17:14:26.426 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:26.415718 I | cephclient: successfully enabled msgr2 protocol 2022-02-21T17:14:26.427 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:26.415762 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-02-21T17:14:26.427 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:26.415770 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-02-21T17:14:26.873 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:26.861660 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-02-21T17:14:26.942 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:26.932431 I | op-mgr: start running mgr 2022-02-21T17:14:26.943 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:26.932469 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-02-21T17:14:27.428 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.418242 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-02-21T17:14:27.443 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.428489 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-02-21T17:14:27.443 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.431977 I | op-mgr: setting services to point to mgr "a" 2022-02-21T17:14:27.587 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.576958 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-02-21T17:14:27.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.577122 I | op-mgr: successful modules: balancer 2022-02-21T17:14:27.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.590951 I | op-osd: start running osds in namespace "rook-ceph" 2022-02-21T17:14:27.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.590999 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-02-21T17:14:27.675 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.663920 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-02-21T17:14:27.717 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705075 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi005-lv-2" 2022-02-21T17:14:27.717 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705119 I | op-osd: OSD PVC "smithi005-lv-2-data-0hnmnc" already exists 2022-02-21T17:14:27.718 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705133 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi005-lv-3" 2022-02-21T17:14:27.718 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705148 I | op-osd: OSD PVC "smithi005-lv-3-data-0glx9j" already exists 2022-02-21T17:14:27.718 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705160 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi005-lv-4" 2022-02-21T17:14:27.718 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705173 I | op-osd: OSD PVC "smithi005-lv-4-data-09b8xq" already exists 2022-02-21T17:14:27.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705185 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi033-lv-1" 2022-02-21T17:14:27.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705198 I | op-osd: OSD PVC "smithi033-lv-1-data-0bqbtn" already exists 2022-02-21T17:14:27.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705210 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi033-lv-2" 2022-02-21T17:14:27.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705223 I | op-osd: OSD PVC "smithi033-lv-2-data-0drpqd" already exists 2022-02-21T17:14:27.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705235 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi033-lv-3" 2022-02-21T17:14:27.720 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705248 I | op-osd: OSD PVC "smithi033-lv-3-data-0t898k" already exists 2022-02-21T17:14:27.720 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705259 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi033-lv-4" 2022-02-21T17:14:27.720 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705271 I | op-osd: OSD PVC "smithi033-lv-4-data-0p96d6" already exists 2022-02-21T17:14:27.720 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705281 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi067-lv-1" 2022-02-21T17:14:27.721 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705296 I | op-osd: OSD PVC "smithi067-lv-1-data-05cnzw" already exists 2022-02-21T17:14:27.721 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705309 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi067-lv-2" 2022-02-21T17:14:27.721 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705321 I | op-osd: OSD PVC "smithi067-lv-2-data-08phkg" already exists 2022-02-21T17:14:27.721 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705332 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi067-lv-3" 2022-02-21T17:14:27.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705344 I | op-osd: OSD PVC "smithi067-lv-3-data-0d6rns" already exists 2022-02-21T17:14:27.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705355 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi067-lv-4" 2022-02-21T17:14:27.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705367 I | op-osd: OSD PVC "smithi067-lv-4-data-0kh9ql" already exists 2022-02-21T17:14:27.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.705378 I | op-osd: creating 1 new PVCs for device set "smithi005-lv-1" 2022-02-21T17:14:27.920 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:27.910048 I | op-osd: successfully provisioned PVC "smithi005-lv-1-data-0qlgp4" 2022-02-21T17:14:28.033 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.019996 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:14:28.033 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020020 I | op-osd: skipping OSD prepare job creation for PVC "smithi005-lv-2-data-0hnmnc" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.033 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020025 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-3-data-0glx9j" 2022-02-21T17:14:28.033 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020028 I | op-osd: skipping OSD prepare job creation for PVC "smithi005-lv-3-data-0glx9j" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.034 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020032 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-4-data-09b8xq" 2022-02-21T17:14:28.034 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020038 I | op-osd: skipping OSD prepare job creation for PVC "smithi005-lv-4-data-09b8xq" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.034 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020044 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:14:28.034 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020065 I | op-osd: skipping OSD prepare job creation for PVC "smithi033-lv-1-data-0bqbtn" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.035 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020069 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-2-data-0drpqd" 2022-02-21T17:14:28.035 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020073 I | op-osd: skipping OSD prepare job creation for PVC "smithi033-lv-2-data-0drpqd" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.035 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020076 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-3-data-0t898k" 2022-02-21T17:14:28.035 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020080 I | op-osd: skipping OSD prepare job creation for PVC "smithi033-lv-3-data-0t898k" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.036 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020083 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-4-data-0p96d6" 2022-02-21T17:14:28.036 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020087 I | op-osd: skipping OSD prepare job creation for PVC "smithi033-lv-4-data-0p96d6" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.036 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020091 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-1-data-05cnzw" 2022-02-21T17:14:28.036 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020095 I | op-osd: skipping OSD prepare job creation for PVC "smithi067-lv-1-data-05cnzw" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.036 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020099 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-2-data-08phkg" 2022-02-21T17:14:28.037 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020103 I | op-osd: skipping OSD prepare job creation for PVC "smithi067-lv-2-data-08phkg" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.037 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020109 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-3-data-0d6rns" 2022-02-21T17:14:28.037 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020115 I | op-osd: skipping OSD prepare job creation for PVC "smithi067-lv-3-data-0d6rns" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.037 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020121 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:14:28.038 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020127 I | op-osd: skipping OSD prepare job creation for PVC "smithi067-lv-4-data-0kh9ql" because OSD daemon using the PVC already exists 2022-02-21T17:14:28.038 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.020133 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-1-data-0qlgp4" 2022-02-21T17:14:28.332 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.321069 I | op-osd: started OSD provisioning job for PVC "smithi005-lv-1-data-0qlgp4" 2022-02-21T17:14:28.332 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.321101 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-02-21T17:14:28.332 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.321108 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-02-21T17:14:28.520 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.510101 I | op-osd: OSD orchestration status for PVC smithi005-lv-1-data-0qlgp4 is "starting" 2022-02-21T17:14:28.622 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.611913 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:28.628 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.618715 I | op-osd: updating OSD 1 on PVC "smithi067-lv-3-data-0d6rns" 2022-02-21T17:14:28.629 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.618750 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-3-data-0d6rns" 2022-02-21T17:14:28.795 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.785541 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:28.802 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.792177 I | op-osd: updating OSD 10 on PVC "smithi005-lv-4-data-09b8xq" 2022-02-21T17:14:28.802 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.792213 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-4-data-09b8xq" 2022-02-21T17:14:28.995 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:28.984947 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:29.014 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:29.004592 I | op-mgr: successful modules: mgr module(s) from the spec 2022-02-21T17:14:29.015 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:29.004780 I | op-mgr: successful modules: dashboard 2022-02-21T17:14:29.017 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:29.005238 I | op-mgr: successful modules: prometheus 2022-02-21T17:14:29.017 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:29.005926 I | op-osd: updating OSD 11 on PVC "smithi033-lv-4-data-0p96d6" 2022-02-21T17:14:29.018 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:29.005947 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-4-data-0p96d6" 2022-02-21T17:14:29.207 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:29.196683 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:29.227 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:29.216873 I | op-osd: updating OSD 2 on PVC "smithi067-lv-1-data-05cnzw" 2022-02-21T17:14:29.227 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:29.216901 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-1-data-05cnzw" 2022-02-21T17:14:29.935 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:29.925216 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:30.028 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:30.017860 I | op-osd: updating OSD 3 on PVC "smithi005-lv-3-data-0glx9j" 2022-02-21T17:14:30.028 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:30.017891 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-3-data-0glx9j" 2022-02-21T17:14:30.625 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:30.614462 I | op-mgr: successful modules: orchestrator modules 2022-02-21T17:14:30.734 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:30.724479 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:30.829 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:30.819448 I | op-osd: updating OSD 4 on PVC "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:14:30.830 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:30.819483 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-2-data-0hnmnc" 2022-02-21T17:14:31.532 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:31.521926 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:31.630 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:31.619710 I | op-osd: updating OSD 5 on PVC "smithi067-lv-2-data-08phkg" 2022-02-21T17:14:31.631 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:31.619750 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-2-data-08phkg" 2022-02-21T17:14:32.338 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:32.324681 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:32.430 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:32.419742 I | op-osd: updating OSD 6 on PVC "smithi033-lv-3-data-0t898k" 2022-02-21T17:14:32.431 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:32.419783 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-3-data-0t898k" 2022-02-21T17:14:32.615 INFO:teuthology.orchestra.run.smithi005.stdout:waiting for osd to come back up 2022-02-21T17:14:32.616 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' 11 '!=' 12 ']' 2022-02-21T17:14:32.617 INFO:teuthology.orchestra.run.smithi005.stderr:+ echo 'waiting for osd to come back up' 2022-02-21T17:14:32.617 INFO:teuthology.orchestra.run.smithi005.stderr:++ toolbox ceph osd stat 2022-02-21T17:14:32.618 INFO:teuthology.orchestra.run.smithi005.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T17:14:32.618 INFO:teuthology.orchestra.run.smithi005.stderr:++ cut -f3 -d ' ' 2022-02-21T17:14:32.705 INFO:teuthology.orchestra.run.smithi005.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T17:14:33.143 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:33.132858 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:33.184 INFO:teuthology.orchestra.run.smithi005.stderr:+ num_osd=11 2022-02-21T17:14:33.184 INFO:teuthology.orchestra.run.smithi005.stderr:+ sleep 30s 2022-02-21T17:14:33.248 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:33.236951 I | op-osd: updating OSD 7 on PVC "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:14:33.248 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:33.236990 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-1-data-0bqbtn" 2022-02-21T17:14:33.937 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:33.926967 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:34.031 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:34.019983 I | op-osd: updating OSD 8 on PVC "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:14:34.031 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:34.020027 I | op-osd: OSD will have its main bluestore block on "smithi067-lv-4-data-0kh9ql" 2022-02-21T17:14:34.755 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:34.744400 I | op-osd: skipping osd checks for ok-to-stop 2022-02-21T17:14:34.835 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:34.824949 I | op-osd: updating OSD 9 on PVC "smithi033-lv-2-data-0drpqd" 2022-02-21T17:14:34.836 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:34.824979 I | op-osd: OSD will have its main bluestore block on "smithi033-lv-2-data-0drpqd" 2022-02-21T17:14:50.224 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:50.213063 I | op-osd: OSD orchestration status for node smithi005-lv-1-data-0qlgp4 is "orchestrating" 2022-02-21T17:14:50.652 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:50.641222 I | op-osd: OSD orchestration status for PVC smithi005-lv-1-data-0qlgp4 is "orchestrating" 2022-02-21T17:14:56.608 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:56.595214 I | op-osd: OSD orchestration status for PVC smithi005-lv-1-data-0qlgp4 is "completed" 2022-02-21T17:14:56.608 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:56.595253 I | op-osd: creating OSD 0 on PVC "smithi005-lv-1-data-0qlgp4" 2022-02-21T17:14:56.609 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:56.595270 I | op-osd: OSD will have its main bluestore block on "smithi005-lv-1-data-0qlgp4" 2022-02-21T17:14:57.728 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:57.716748 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-02-21T17:14:57.729 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:14:57.716781 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-02-21T17:15:03.187 INFO:teuthology.orchestra.run.smithi005.stdout:waiting for osd to come back up 2022-02-21T17:15:03.188 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' 11 '!=' 12 ']' 2022-02-21T17:15:03.188 INFO:teuthology.orchestra.run.smithi005.stderr:+ echo 'waiting for osd to come back up' 2022-02-21T17:15:03.189 INFO:teuthology.orchestra.run.smithi005.stderr:++ toolbox ceph osd stat 2022-02-21T17:15:03.189 INFO:teuthology.orchestra.run.smithi005.stderr:++ cut -f3 -d ' ' 2022-02-21T17:15:03.189 INFO:teuthology.orchestra.run.smithi005.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T17:15:03.254 INFO:teuthology.orchestra.run.smithi005.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T17:15:03.746 INFO:teuthology.orchestra.run.smithi005.stderr:+ num_osd=11 2022-02-21T17:15:03.747 INFO:teuthology.orchestra.run.smithi005.stderr:+ sleep 30s 2022-02-21T17:15:33.749 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' 11 '!=' 12 ']' 2022-02-21T17:15:33.749 INFO:teuthology.orchestra.run.smithi005.stdout:waiting for osd to come back up 2022-02-21T17:15:33.750 INFO:teuthology.orchestra.run.smithi005.stderr:+ echo 'waiting for osd to come back up' 2022-02-21T17:15:33.750 INFO:teuthology.orchestra.run.smithi005.stderr:++ toolbox ceph osd stat 2022-02-21T17:15:33.750 INFO:teuthology.orchestra.run.smithi005.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-02-21T17:15:33.751 INFO:teuthology.orchestra.run.smithi005.stderr:++ cut -f3 -d ' ' 2022-02-21T17:15:33.824 INFO:teuthology.orchestra.run.smithi005.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-02-21T17:15:34.336 INFO:teuthology.orchestra.run.smithi005.stderr:+ num_osd=12 2022-02-21T17:15:34.337 INFO:teuthology.orchestra.run.smithi005.stderr:+ sleep 30s 2022-02-21T17:16:04.338 INFO:teuthology.orchestra.run.smithi005.stderr:+ '[' 12 '!=' 12 ']' 2022-02-21T17:16:04.340 INFO:teuthology.run_tasks:Running task rook.shell... 2022-02-21T17:16:04.354 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch status 2022-02-21T17:16:05.131 INFO:teuthology.orchestra.run.smithi005.stdout:Backend: rook 2022-02-21T17:16:05.131 INFO:teuthology.orchestra.run.smithi005.stdout:Available: Yes 2022-02-21T17:16:05.154 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ps 2022-02-21T17:16:05.713 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-02-21T17:16:05.713 INFO:teuthology.orchestra.run.smithi005.stdout:crashcollector.crash smithi005 running (5m) 0s ago 5m - - f61e120f25ae 2022-02-21T17:16:05.714 INFO:teuthology.orchestra.run.smithi005.stdout:crashcollector.crash smithi033 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:16:05.714 INFO:teuthology.orchestra.run.smithi005.stdout:crashcollector.crash smithi067 running (5m) 0s ago 5m - - f61e120f25ae 2022-02-21T17:16:05.714 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.a smithi033 running (5m) 0s ago 5m - - f61e120f25ae 2022-02-21T17:16:05.714 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi067 running (7m) 0s ago 7m - - f61e120f25ae 2022-02-21T17:16:05.715 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi033 running (7m) 0s ago 7m - - f61e120f25ae 2022-02-21T17:16:05.715 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (6m) 0s ago 6m - - f61e120f25ae 2022-02-21T17:16:05.715 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (68s) 0s ago 68s - - f61e120f25ae 2022-02-21T17:16:05.715 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi067 running (4m) 0s ago 4m - - f61e120f25ae 2022-02-21T17:16:05.716 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10 smithi005 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:16:05.716 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11 smithi033 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:16:05.716 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi067 running (4m) 0s ago 4m - - f61e120f25ae 2022-02-21T17:16:05.716 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (4m) 0s ago 4m - - f61e120f25ae 2022-02-21T17:16:05.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi005 running (4m) 0s ago 4m - - f61e120f25ae 2022-02-21T17:16:05.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi067 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:16:05.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi033 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:16:05.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi033 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:16:05.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8 smithi067 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:16:05.718 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9 smithi033 running (3m) 0s ago 3m - - f61e120f25ae 2022-02-21T17:16:05.736 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls 2022-02-21T17:16:06.415 INFO:teuthology.orchestra.run.smithi005.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-02-21T17:16:06.415 INFO:teuthology.orchestra.run.smithi005.stdout:crash 3/3 0s ago 5m * 2022-02-21T17:16:06.416 INFO:teuthology.orchestra.run.smithi005.stdout:mgr 1/1 0s ago 5m count:1 2022-02-21T17:16:06.416 INFO:teuthology.orchestra.run.smithi005.stdout:mon 3/3 0s ago 7m count:3 2022-02-21T17:16:06.416 INFO:teuthology.orchestra.run.smithi005.stdout:osd 24 0s ago 4m 2022-02-21T17:16:06.416 INFO:teuthology.orchestra.run.smithi005.stdout:osd.all-available-devices 0 0s ago - * 2022-02-21T17:16:06.439 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch host ls 2022-02-21T17:16:06.990 INFO:teuthology.orchestra.run.smithi005.stdout:HOST ADDR LABELS STATUS 2022-02-21T17:16:06.990 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 172.21.15.5/smithi005 2022-02-21T17:16:06.991 INFO:teuthology.orchestra.run.smithi005.stdout:smithi033 172.21.15.33/smithi033 2022-02-21T17:16:06.991 INFO:teuthology.orchestra.run.smithi005.stdout:smithi067 172.21.15.67/smithi067 2022-02-21T17:16:06.991 INFO:teuthology.orchestra.run.smithi005.stdout:3 hosts in cluster 2022-02-21T17:16:07.012 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch device ls 2022-02-21T17:16:07.586 INFO:teuthology.orchestra.run.smithi005.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2022-02-21T17:16:07.586 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T17:16:07.587 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T17:16:07.587 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T17:16:07.587 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T17:16:07.587 INFO:teuthology.orchestra.run.smithi005.stdout:smithi033 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T17:16:07.587 INFO:teuthology.orchestra.run.smithi005.stdout:smithi033 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T17:16:07.588 INFO:teuthology.orchestra.run.smithi005.stdout:smithi033 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T17:16:07.588 INFO:teuthology.orchestra.run.smithi005.stdout:smithi033 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T17:16:07.588 INFO:teuthology.orchestra.run.smithi005.stdout:smithi067 /dev/vg_nvme/lv_1 unknown None 107G 2022-02-21T17:16:07.588 INFO:teuthology.orchestra.run.smithi005.stdout:smithi067 /dev/vg_nvme/lv_2 unknown None 107G 2022-02-21T17:16:07.588 INFO:teuthology.orchestra.run.smithi005.stdout:smithi067 /dev/vg_nvme/lv_3 unknown None 107G 2022-02-21T17:16:07.589 INFO:teuthology.orchestra.run.smithi005.stdout:smithi067 /dev/vg_nvme/lv_4 unknown None 107G 2022-02-21T17:16:07.617 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch apply rgw foo 2022-02-21T17:16:08.176 INFO:teuthology.orchestra.run.smithi005.stdout:Created 2022-02-21T17:16:08.180 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:08.168006 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-02-21T17:16:08.204 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch apply mds foo 2022-02-21T17:16:08.285 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:08.271725 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:08.286 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:08.271799 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T17:16:08.800 INFO:teuthology.orchestra.run.smithi005.stdout:Created 2022-02-21T17:16:08.802 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:08.789639 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-02-21T17:16:08.822 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch apply rbd-mirror 2022-02-21T17:16:08.843 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:08.830332 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:08.844 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:08.830403 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T17:16:09.449 INFO:teuthology.orchestra.run.smithi005.stdout:Success 2022-02-21T17:16:09.471 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch apply nfs foo --port 12777 2022-02-21T17:16:09.503 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:09.490677 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:09.503 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:09.490764 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T17:16:13.570 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:13.557110 I | ceph-spec: adding finalizer "cephnfs.ceph.rook.io" on "foo" 2022-02-21T17:16:13.571 INFO:teuthology.orchestra.run.smithi005.stdout:Created 2022-02-21T17:16:13.598 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-02-21T17:16:13.611 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:16:13.641 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:13.628624 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:13.642 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:13.628683 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e... 2022-02-21T17:16:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:16:14.441 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:44.000000Z", "last_refresh": "2022-02-21T17:16:14.255672Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:14.255672Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:16:14.255672Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:16:14.255672Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "status": {"created": "2022-02-21T17:16:13.000000Z", "last_refresh": "2022-02-21T17:16:14.255672Z", "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:16:14.255672Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:16:14.255672Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-02-21T17:16:14.255672Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:14.255672Z", "running": 0, "size": 1}}] 2022-02-21T17:16:14.460 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch rm mds.foo 2022-02-21T17:16:16.161 INFO:teuthology.orchestra.run.smithi005.stdout:Removed cephfilesystems/foo 2022-02-21T17:16:16.184 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch rm nfs.foo 2022-02-21T17:16:17.120 INFO:teuthology.orchestra.run.smithi005.stdout:Removed cephnfses/foo 2022-02-21T17:16:17.141 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch rm rbd-mirror 2022-02-21T17:16:17.630 INFO:teuthology.orchestra.run.smithi005.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-02-21T17:16:17.647 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch rm rgw.foo 2022-02-21T17:16:18.217 INFO:teuthology.orchestra.run.smithi005.stdout:Removed cephobjectstores/foo 2022-02-21T17:16:18.238 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:16:18.738 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:18.725690 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T17:16:18.896 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:18.883852 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T17:16:18.899 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:18.885945 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T17:16:19.200 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:19.144904 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-02-21T17:16:19.200 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:19.144938 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-02-21T17:16:19.201 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:19.144954 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-02-21T17:16:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:16:19.228 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:44.000000Z", "last_refresh": "2022-02-21T17:16:18.822868Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:18.822868Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:16:18.822868Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:16:18.822868Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "status": {"created": "2022-02-21T17:16:13.000000Z", "last_refresh": "2022-02-21T17:16:18.822868Z", "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:16:18.822868Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:16:18.822868Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:18.822868Z", "running": 0, "size": 1}}] 2022-02-21T17:16:20.174 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:20.162140 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-02-21T17:16:20.242 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:20.230049 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-02-21T17:16:20.353 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:20.340088 I | ceph-object-controller: reconciling object store deployments 2022-02-21T17:16:20.470 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:20.457267 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-02-21T17:16:20.498 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:20.482179 I | ceph-object-controller: ceph object store gateway service running at 10.105.2.201 2022-02-21T17:16:20.498 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:20.482202 I | ceph-object-controller: reconciling object store pools 2022-02-21T17:16:20.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:20.588182 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi067": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T17:16:20.944 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:20.931267 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-02-21T17:16:20.967 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:20.954362 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-02-21T17:16:22.002 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:21.989647 I | op-mds: setting mds config flags 2022-02-21T17:16:22.003 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:21.989664 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-02-21T17:16:23.308 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:23.294972 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-02-21T17:16:23.322 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:23.309253 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-02-21T17:16:25.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:25.714602 I | op-mds: setting mds config flags 2022-02-21T17:16:25.728 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:25.714637 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-02-21T17:16:26.055 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:26.041448 I | ceph-spec: detected ceph image version: "17.0.0-10781 quincy" 2022-02-21T17:16:26.451 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:26.438420 I | ceph-nfs-controller: configuring pool ".nfs" for nfs 2022-02-21T17:16:28.112 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:28.099076 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-02-21T17:16:29.255 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:16:29.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:29.483079 I | ceph-file-controller: creating filesystem "foo" 2022-02-21T17:16:30.368 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:16:30.369 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:44.000000Z", "last_refresh": "2022-02-21T17:16:30.173489Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:30.173489Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:16:30.173489Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:16:30.173489Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "status": {"created": "2022-02-21T17:16:13.000000Z", "last_refresh": "2022-02-21T17:16:30.173489Z", "running": 0, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:16:30.173489Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:16:30.173489Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:30.173489Z", "running": 0, "size": 1}}] 2022-02-21T17:16:31.112 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:31.099763 I | ceph-nfs-controller: set pool ".nfs" for the application nfs 2022-02-21T17:16:31.119 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:31.106959 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-02-21T17:16:31.120 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:31.107078 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-02-21T17:16:31.125 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:31.111106 I | ceph-nfs-controller: updating ceph nfs "foo" 2022-02-21T17:16:31.343 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:31.330873 I | cephclient: getting or creating ceph auth key "client.nfs-ganesha.foo.a" 2022-02-21T17:16:32.328 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:32.315285 I | ceph-nfs-controller: ceph nfs deployment "rook-ceph-nfs-foo-a" started 2022-02-21T17:16:32.483 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:32.435638 I | ceph-nfs-controller: ceph nfs service running at 10.103.90.210:2049 2022-02-21T17:16:32.483 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:32.435675 I | ceph-nfs-controller: adding ganesha "a" to grace db 2022-02-21T17:16:32.574 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:32.561697 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:32.575 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:32.561780 I | ceph-nfs-controller: deleting ceph nfs "foo" 2022-02-21T17:16:32.846 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:32.832886 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-02-21T17:16:32.848 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:32.834876 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-02-21T17:16:32.854 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:32.841655 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-02-21T17:16:33.499 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:33.486568 I | ceph-nfs-controller: removing ganesha "a" from grace db 2022-02-21T17:16:33.701 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:33.688205 I | ceph-spec: removing finalizer "cephnfs.ceph.rook.io" on "foo" 2022-02-21T17:16:33.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:33.714317 I | ceph-spec: object "rook-ceph-nfs-foo-a" matched on delete, reconciling 2022-02-21T17:16:33.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:33.714504 I | ceph-spec: object "rook-ceph-nfs-foo-a" matched on delete, reconciling 2022-02-21T17:16:33.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:33.714585 I | ceph-spec: object "rook-ceph-nfs-foo-a" matched on delete, reconciling 2022-02-21T17:16:34.760 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:34.747618 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-02-21T17:16:34.771 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:34.758173 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:35.724 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:35.597398 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-02-21T17:16:39.271 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:39.257457 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:39.250+0000 7f6ce1113340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:39.271 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:39.250+0000 7f6ce1113340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:39.271 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:39.250+0000 7f6ce1113340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:39.272 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:39.285 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:39.272466 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:40.055 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:40.042260 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:40.034+0000 7fc63156e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:40.055 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:40.034+0000 7fc63156e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:40.056 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:40.034+0000 7fc63156e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:40.056 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:40.092 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:40.079308 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:40.395 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:16:41.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:41.012169 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-02-21T17:16:41.096 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:16:41.097 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:16:40.903785Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:40.903785Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:16:40.903785Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:16:40.903785Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:16:40.903785Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:16:40.903785Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:40.903785Z", "running": 0, "size": 1}}] 2022-02-21T17:16:41.276 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:41.262657 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:41.254+0000 7f9aa8036340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:41.276 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:41.254+0000 7f9aa8036340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:41.277 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:41.254+0000 7f9aa8036340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:41.277 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:41.285 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:41.272783 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:41.851 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:41.838129 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-02-21T17:16:41.874 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:41.861383 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:41.854+0000 7f17c399a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:41.875 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:41.854+0000 7f17c399a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:41.875 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:41.854+0000 7f17c399a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:41.875 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:41.885 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:41.872581 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:42.469 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:42.456132 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:42.450+0000 7f5783423340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:42.469 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:42.450+0000 7f5783423340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:42.469 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:42.450+0000 7f5783423340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:42.470 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:42.485 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:42.472973 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:43.065 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:43.052705 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:43.046+0000 7fd62b05d340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:43.066 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:43.046+0000 7fd62b05d340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:43.066 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:43.046+0000 7fd62b05d340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:43.066 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:43.077 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:43.064755 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:43.660 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:43.647360 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:43.638+0000 7fa7cdc7e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:43.661 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:43.638+0000 7fa7cdc7e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:43.661 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:43.638+0000 7fa7cdc7e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:43.661 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:43.676 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:43.664193 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:43.896 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:43.883708 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-02-21T17:16:43.897 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:43.883730 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-02-21T17:16:44.279 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:44.265925 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:44.258+0000 7fbea8111340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:44.279 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:44.258+0000 7fbea8111340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:44.280 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:44.258+0000 7fbea8111340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:44.280 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:44.306 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:44.293377 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:44.462 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:44.449553 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:44.466 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:44.451828 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-02-21T17:16:45.000 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:44.987030 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:44.978+0000 7fcdd291b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:45.001 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:44.978+0000 7fcdd291b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:45.001 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:44.978+0000 7fcdd291b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:45.002 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:45.447 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:45.433956 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-02-21T17:16:45.945 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:45.932007 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-02-21T17:16:45.945 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:45.932030 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-02-21T17:16:45.945 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:45.932036 I | cephclient: deleting ceph auth "mds.foo-a" 2022-02-21T17:16:46.465 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:46.452221 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-02-21T17:16:46.821 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:46.808028 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-02-21T17:16:47.263 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:47.250005 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-02-21T17:16:47.264 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:47.250038 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-02-21T17:16:47.264 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:47.250048 I | cephclient: deleting ceph auth "mds.foo-b" 2022-02-21T17:16:47.573 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:47.560464 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:47.756 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:47.743592 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-02-21T17:16:47.757 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:47.743635 I | ceph-file-controller: downing filesystem "foo" 2022-02-21T17:16:48.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:48.170951 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:48.162+0000 7f32c9970340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:48.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:48.162+0000 7f32c9970340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:48.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:48.162+0000 7f32c9970340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:48.185 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:48.375 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:48.362126 I | ceph-file-controller: downed filesystem "foo" 2022-02-21T17:16:51.119 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:16:51.852 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:16:51.852 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:16:51.658422Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:51.658422Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:16:51.658422Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:16:51.658422Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:16:51.658422Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:16:51.658422Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:16:51.658422Z", "running": 0, "size": 1}}] 2022-02-21T17:16:52.508 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:52.494579 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-02-21T17:16:52.509 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:52.494601 I | cephclient: purging pool "foo-metadata" (id=8) 2022-02-21T17:16:53.312 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:53.299364 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:16:53.902 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:53.888766 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:16:53.881+0000 7fdf226d5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:16:53.902 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:53.881+0000 7fdf226d5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:16:53.903 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:16:53.881+0000 7fdf226d5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:16:53.903 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:16:54.868 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:54.854888 I | cephclient: purge completed for pool "foo-metadata" 2022-02-21T17:16:55.295 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:55.282736 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-02-21T17:16:55.296 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:55.282757 I | cephclient: purging pool "foo-data0" (id=10) 2022-02-21T17:16:57.251 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:57.238022 I | cephclient: purge completed for pool "foo-data0" 2022-02-21T17:16:57.251 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:57.238105 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-02-21T17:16:57.292 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:57.278633 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-02-21T17:16:57.301 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:57.287675 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-02-21T17:16:57.321 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:57.307562 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-02-21T17:16:57.386 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:16:57.373849 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-02-21T17:17:01.882 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:17:02.683 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:17:02.684 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:17:02.461711Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:17:02.461711Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:17:02.461711Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:17:02.461711Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:17:02.461711Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:17:02.461711Z", "running": 0, "size": 1}}] 2022-02-21T17:17:04.148 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:17:04.135286 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:17:04.721 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:17:04.708128 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:17:04.697+0000 7f1e8193a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:17:04.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:17:04.697+0000 7f1e8193a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:17:04.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:17:04.697+0000 7f1e8193a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:17:04.722 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:17:12.709 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:17:13.472 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:17:13.473 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:17:13.226239Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:17:13.226239Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:17:13.226239Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:17:13.226239Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:17:13.226239Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:17:13.226239Z", "running": 0, "size": 1}}] 2022-02-21T17:17:23.495 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:17:25.210 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:17:25.196639 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:17:25.877 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:17:25.863357 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:17:25.853+0000 7fcc505ef340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:17:25.877 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:17:25.853+0000 7fcc505ef340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:17:25.877 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:17:25.853+0000 7fcc505ef340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:17:25.878 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:17:26.719 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:17:26.720 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:17:26.523267Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:17:26.523267Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:17:26.523267Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:17:26.523267Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:17:26.523267Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:17:26.523267Z", "running": 0, "size": 1}}] 2022-02-21T17:17:36.747 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:17:37.419 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:17:37.420 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:17:37.233861Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:17:37.233861Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:17:37.233861Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:17:37.233861Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:17:37.233861Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:17:37.233861Z", "running": 0, "size": 1}}] 2022-02-21T17:17:47.447 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:17:48.141 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:17:48.141 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:17:47.956374Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:17:47.956374Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:17:47.956374Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:17:47.956374Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:17:47.956374Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:17:47.956374Z", "running": 0, "size": 1}}] 2022-02-21T17:17:58.169 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:17:58.942 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:17:58.943 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:17:58.708364Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:17:58.708364Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:17:58.708364Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:17:58.708364Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:17:58.708364Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:17:58.708364Z", "running": 0, "size": 1}}] 2022-02-21T17:18:06.848 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:18:06.833288 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:18:07.522 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:18:07.508539 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:18:07.495+0000 7f3a11581340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:18:07.523 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:18:07.495+0000 7f3a11581340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:18:07.523 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:18:07.495+0000 7f3a11581340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:18:07.523 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:18:08.966 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:18:09.703 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:18:09.703 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:18:09.512942Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:18:09.512942Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:18:09.512942Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:18:09.512942Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:18:09.512942Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:18:09.512942Z", "running": 0, "size": 1}}] 2022-02-21T17:18:19.731 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:18:20.545 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:18:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:18:20.286306Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:18:20.286306Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:18:20.286306Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:18:20.286306Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:18:20.286306Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:18:20.286306Z", "running": 0, "size": 1}}] 2022-02-21T17:18:24.960 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:18:24.946898 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi067": the object has been modified; please apply your changes to the latest version and try again 2022-02-21T17:18:30.571 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:18:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:18:31.307 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:18:31.077792Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:18:31.077792Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:18:31.077792Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:18:31.077792Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:18:31.077792Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:18:31.077792Z", "running": 0, "size": 1}}] 2022-02-21T17:18:41.334 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:18:42.060 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:18:42.061 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:18:41.869524Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:18:41.869524Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:18:41.869524Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:18:41.869524Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:18:41.869524Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:18:41.869524Z", "running": 0, "size": 1}}] 2022-02-21T17:18:52.086 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:18:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:18:52.818 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:18:52.631371Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:18:52.631371Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:18:52.631371Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:18:52.631371Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:18:52.631371Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:18:52.631371Z", "running": 0, "size": 1}}] 2022-02-21T17:19:02.844 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:19:03.628 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:19:03.629 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:19:03.392334Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:19:03.392334Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:19:03.392334Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:19:03.392334Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:19:03.392334Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:19:03.392334Z", "running": 0, "size": 1}}] 2022-02-21T17:19:13.656 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:19:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:19:14.471 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:19:14.230618Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:19:14.230618Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:19:14.230618Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:19:14.230618Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:19:14.230618Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:19:14.230618Z", "running": 0, "size": 1}}] 2022-02-21T17:19:24.498 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:19:25.315 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:19:25.315 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:19:25.120707Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:19:25.120707Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:19:25.120707Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:19:25.120707Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:19:25.120707Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:19:25.120707Z", "running": 0, "size": 1}}] 2022-02-21T17:19:29.449 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:19:29.434996 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:19:30.129 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:19:30.113949 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:19:30.101+0000 7f52bae41340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:19:30.129 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:19:30.101+0000 7f52bae41340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:19:30.129 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:19:30.101+0000 7f52bae41340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:19:30.129 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:19:35.343 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:19:36.027 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:19:36.028 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:19:35.842444Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:19:35.842444Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:19:35.842444Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:19:35.842444Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:19:35.842444Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:19:35.842444Z", "running": 0, "size": 1}}] 2022-02-21T17:19:46.057 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:19:46.766 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:19:46.766 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:19:46.576088Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:19:46.576088Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:19:46.576088Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:19:46.576088Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:19:46.576088Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:19:46.576088Z", "running": 0, "size": 1}}] 2022-02-21T17:19:56.803 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:19:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:19:57.528 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:19:57.338905Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:19:57.338905Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:19:57.338905Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:19:57.338905Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:19:57.338905Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:19:57.338905Z", "running": 0, "size": 1}}] 2022-02-21T17:20:07.556 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:20:08.318 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:20:08.318 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:20:08.116102Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:20:08.116102Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:20:08.116102Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:20:08.116102Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:20:08.116102Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:20:08.116102Z", "running": 0, "size": 1}}] 2022-02-21T17:20:18.342 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:20:19.070 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:20:19.070 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:20:18.874938Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:20:18.874938Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:20:18.874938Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:20:18.874938Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:20:18.874938Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:20:18.874938Z", "running": 0, "size": 1}}] 2022-02-21T17:20:29.093 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:20:29.853 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:20:29.854 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:20:29.616946Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:20:29.616946Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:20:29.616946Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:20:29.616946Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:20:29.616946Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:20:29.616946Z", "running": 0, "size": 1}}] 2022-02-21T17:20:39.881 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:20:40.625 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:20:40.626 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:20:40.425736Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:20:40.425736Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:20:40.425736Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:20:40.425736Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:20:40.425736Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:20:40.425736Z", "running": 0, "size": 1}}] 2022-02-21T17:20:50.653 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:20:51.412 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:20:51.413 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:20:51.202980Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:20:51.202980Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:20:51.202980Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:20:51.202980Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:20:51.202980Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:20:51.202980Z", "running": 0, "size": 1}}] 2022-02-21T17:21:01.440 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:21:02.176 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:21:02.176 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:21:01.983946Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:21:01.983946Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:21:01.983946Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:21:01.983946Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:21:01.983946Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:21:01.983946Z", "running": 0, "size": 1}}] 2022-02-21T17:21:12.203 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:21:12.883 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:21:12.884 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:21:12.705807Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:21:12.705807Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:21:12.705807Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:21:12.705807Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:21:12.705807Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:21:12.705807Z", "running": 0, "size": 1}}] 2022-02-21T17:21:22.910 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:21:23.669 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:21:23.669 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:21:23.456383Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:21:23.456383Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:21:23.456383Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:21:23.456383Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:21:23.456383Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:21:23.456383Z", "running": 0, "size": 1}}] 2022-02-21T17:21:33.695 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:21:34.466 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:21:34.466 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:21:34.222113Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:21:34.222113Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:21:34.222113Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:21:34.222113Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:21:34.222113Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:21:34.222113Z", "running": 0, "size": 1}}] 2022-02-21T17:21:44.493 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:21:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:21:45.275 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:21:45.080937Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:21:45.080937Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:21:45.080937Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:21:45.080937Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:21:45.080937Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:21:45.080937Z", "running": 0, "size": 1}}] 2022-02-21T17:21:55.300 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:21:55.991 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:21:55.992 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:21:55.798954Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:21:55.798954Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:21:55.798954Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:21:55.798954Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:21:55.798954Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:21:55.798954Z", "running": 0, "size": 1}}] 2022-02-21T17:22:06.021 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:22:06.799 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:22:06.799 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:22:06.552426Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:22:06.552426Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:22:06.552426Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:22:06.552426Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:22:06.552426Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:22:06.552426Z", "running": 0, "size": 1}}] 2022-02-21T17:22:13.978 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:22:13.962410 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:22:14.684 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:22:14.668068 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:22:14.656+0000 7fc27ff5d340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:22:14.685 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:22:14.656+0000 7fc27ff5d340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:22:14.685 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:22:14.656+0000 7fc27ff5d340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:22:14.685 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:22:16.829 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:22:17.541 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:22:17.541 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:22:17.347487Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:22:17.347487Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:22:17.347487Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:22:17.347487Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:22:17.347487Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:22:17.347487Z", "running": 0, "size": 1}}] 2022-02-21T17:22:27.567 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:22:28.316 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:22:28.317 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:22:28.126093Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:22:28.126093Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:22:28.126093Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:22:28.126093Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:22:28.126093Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:22:28.126093Z", "running": 0, "size": 1}}] 2022-02-21T17:22:38.344 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:22:39.074 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:22:39.074 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:22:38.898205Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:22:38.898205Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:22:38.898205Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:22:38.898205Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:22:38.898205Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:22:38.898205Z", "running": 0, "size": 1}}] 2022-02-21T17:22:49.102 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:22:49.807 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:22:49.808 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:22:49.615264Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:22:49.615264Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:22:49.615264Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:22:49.615264Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:22:49.615264Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:22:49.615264Z", "running": 0, "size": 1}}] 2022-02-21T17:22:59.835 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:23:00.595 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:23:00.596 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:23:00.405251Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:23:00.405251Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:23:00.405251Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:23:00.405251Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:23:00.405251Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:23:00.405251Z", "running": 0, "size": 1}}] 2022-02-21T17:23:10.623 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:23:11.373 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:23:11.373 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:23:11.161833Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:23:11.161833Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:23:11.161833Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:23:11.161833Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:23:11.161833Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:23:11.161833Z", "running": 0, "size": 1}}] 2022-02-21T17:23:21.401 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:23:22.114 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:23:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:23:21.901816Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:23:21.901816Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:23:21.901816Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:23:21.901816Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:23:21.901816Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:23:21.901816Z", "running": 0, "size": 1}}] 2022-02-21T17:23:32.141 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:23:32.848 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:23:32.849 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:23:32.661027Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:23:32.661027Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:23:32.661027Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:23:32.661027Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:23:32.661027Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:23:32.661027Z", "running": 0, "size": 1}}] 2022-02-21T17:23:42.881 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:23:43.603 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:23:43.603 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:23:43.406015Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:23:43.406015Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:23:43.406015Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:23:43.406015Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:23:43.406015Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:23:43.406015Z", "running": 0, "size": 1}}] 2022-02-21T17:23:53.632 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:23:54.370 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:23:54.371 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:23:54.147652Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:23:54.147652Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:23:54.147652Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:23:54.147652Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:23:54.147652Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:23:54.147652Z", "running": 0, "size": 1}}] 2022-02-21T17:24:04.397 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:24:05.143 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:24:05.143 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:24:04.938250Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:24:04.938250Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:24:04.938250Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:24:04.938250Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:24:04.938250Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:24:04.938250Z", "running": 0, "size": 1}}] 2022-02-21T17:24:15.167 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:24:16.159 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:24:16.159 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:24:15.675579Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:24:15.675579Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:24:15.675579Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:24:15.675579Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:24:15.675579Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:24:15.675579Z", "running": 0, "size": 1}}] 2022-02-21T17:24:26.185 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:24:26.931 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:24:26.931 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:24:26.733212Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:24:26.733212Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:24:26.733212Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:24:26.733212Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:24:26.733212Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:24:26.733212Z", "running": 0, "size": 1}}] 2022-02-21T17:24:36.968 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:24:37.706 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:24:37.706 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:24:37.523932Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:24:37.523932Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:24:37.523932Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:24:37.523932Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:24:37.523932Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:24:37.523932Z", "running": 0, "size": 1}}] 2022-02-21T17:24:47.733 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:24:48.466 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:24:48.467 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:24:48.261755Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:24:48.261755Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:24:48.261755Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:24:48.261755Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:24:48.261755Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:24:48.261755Z", "running": 0, "size": 1}}] 2022-02-21T17:24:58.494 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:24:59.261 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:24:59.262 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:24:59.023914Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:24:59.023914Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:24:59.023914Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:24:59.023914Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:24:59.023914Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:24:59.023914Z", "running": 0, "size": 1}}] 2022-02-21T17:25:09.287 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:25:09.999 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:25:09.999 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:25:09.790861Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:25:09.790861Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:25:09.790861Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:25:09.790861Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:25:09.790861Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:25:09.790861Z", "running": 0, "size": 1}}] 2022-02-21T17:25:20.025 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:25:20.750 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:25:20.751 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:25:20.550864Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:25:20.550864Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:25:20.550864Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:25:20.550864Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:25:20.550864Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:25:20.550864Z", "running": 0, "size": 1}}] 2022-02-21T17:25:30.778 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:25:31.557 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:25:31.725 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:25:31.281524Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:25:31.281524Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:25:31.281524Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:25:31.281524Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:25:31.281524Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:25:31.281524Z", "running": 0, "size": 1}}] 2022-02-21T17:25:41.728 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:25:42.458 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:25:42.459 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:25:42.273367Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:25:42.273367Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:25:42.273367Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:25:42.273367Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:25:42.273367Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:25:42.273367Z", "running": 0, "size": 1}}] 2022-02-21T17:25:52.486 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:25:53.184 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:25:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:25:52.991184Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:25:52.991184Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:25:52.991184Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:25:52.991184Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:25:52.991184Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:25:52.991184Z", "running": 0, "size": 1}}] 2022-02-21T17:26:03.209 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:26:03.959 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:26:03.959 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:26:03.772164Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:26:03.772164Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:26:03.772164Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:26:03.772164Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:26:03.772164Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:26:03.772164Z", "running": 0, "size": 1}}] 2022-02-21T17:26:13.988 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:26:14.707 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:26:14.707 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:26:14.512212Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:26:14.512212Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:26:14.512212Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:26:14.512212Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:26:14.512212Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:26:14.512212Z", "running": 0, "size": 1}}] 2022-02-21T17:26:24.728 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:26:25.502 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:26:25.503 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:26:25.294036Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:26:25.294036Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:26:25.294036Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:26:25.294036Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:26:25.294036Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:26:25.294036Z", "running": 0, "size": 1}}] 2022-02-21T17:26:35.530 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:26:36.245 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:26:36.245 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:26:36.049191Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:26:36.049191Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:26:36.049191Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:26:36.049191Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:26:36.049191Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:26:36.049191Z", "running": 0, "size": 1}}] 2022-02-21T17:26:46.272 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:26:47.029 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:26:47.030 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:26:46.820833Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:26:46.820833Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:26:46.820833Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:26:46.820833Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:26:46.820833Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:26:46.820833Z", "running": 0, "size": 1}}] 2022-02-21T17:26:57.062 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:26:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:26:57.822 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:26:57.619136Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:26:57.619136Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:26:57.619136Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:26:57.619136Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:26:57.619136Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:26:57.619136Z", "running": 0, "size": 1}}] 2022-02-21T17:27:07.851 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:27:08.591 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:27:08.592 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:27:08.398673Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:27:08.398673Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:27:08.398673Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:27:08.398673Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:27:08.398673Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:27:08.398673Z", "running": 0, "size": 1}}] 2022-02-21T17:27:18.619 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:27:19.353 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:27:19.353 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:27:19.145605Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:27:19.145605Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:27:19.145605Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:27:19.145605Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:27:19.145605Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:27:19.145605Z", "running": 0, "size": 1}}] 2022-02-21T17:27:29.380 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:27:32.355 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:27:32.356 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:27:32.135710Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:27:32.135710Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:27:32.135710Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:27:32.135710Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:27:32.135710Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:27:32.135710Z", "running": 0, "size": 1}}] 2022-02-21T17:27:42.373 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:27:42.355433 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:27:42.382 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:27:43.079 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:27:43.060215 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:27:43.049+0000 7f823d6a8340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:27:43.079 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:27:43.049+0000 7f823d6a8340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:27:43.079 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:27:43.049+0000 7f823d6a8340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:27:43.080 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:27:43.095 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:27:43.096 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:27:42.908224Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:27:42.908224Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:27:42.908224Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:27:42.908224Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:27:42.908224Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:27:42.908224Z", "running": 0, "size": 1}}] 2022-02-21T17:27:53.121 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:27:53.883 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:27:53.883 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:27:53.678994Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:27:53.678994Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:27:53.678994Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:27:53.678994Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:27:53.678994Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:27:53.678994Z", "running": 0, "size": 1}}] 2022-02-21T17:28:03.909 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:28:04.726 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:28:04.726 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:28:04.434298Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:28:04.434298Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:28:04.434298Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:28:04.434298Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:28:04.434298Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:28:04.434298Z", "running": 0, "size": 1}}] 2022-02-21T17:28:14.750 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:28:15.504 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:28:15.504 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:28:15.280407Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:28:15.280407Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:28:15.280407Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:28:15.280407Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:28:15.280407Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:28:15.280407Z", "running": 0, "size": 1}}] 2022-02-21T17:28:25.531 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:28:26.242 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:28:26.243 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:28:26.049581Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:28:26.049581Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:28:26.049581Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:28:26.049581Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:28:26.049581Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:28:26.049581Z", "running": 0, "size": 1}}] 2022-02-21T17:28:36.269 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:28:36.998 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:28:36.998 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:28:36.789434Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:28:36.789434Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:28:36.789434Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:28:36.789434Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:28:36.789434Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:28:36.789434Z", "running": 0, "size": 1}}] 2022-02-21T17:28:47.031 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:28:47.810 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:28:47.810 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:28:47.609431Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:28:47.609431Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:28:47.609431Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:28:47.609431Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:28:47.609431Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:28:47.609431Z", "running": 0, "size": 1}}] 2022-02-21T17:28:57.837 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:28:58.622 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:28:58.623 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:28:58.407775Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:28:58.407775Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:28:58.407775Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:28:58.407775Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:28:58.407775Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:28:58.407775Z", "running": 0, "size": 1}}] 2022-02-21T17:29:08.652 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:29:09.406 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:29:09.406 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:29:09.166239Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:29:09.166239Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:29:09.166239Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:29:09.166239Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:29:09.166239Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:29:09.166239Z", "running": 0, "size": 1}}] 2022-02-21T17:29:19.427 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:29:20.120 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:29:20.120 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:29:19.918377Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:29:19.918377Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:29:19.918377Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:29:19.918377Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:29:19.918377Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:29:19.918377Z", "running": 0, "size": 1}}] 2022-02-21T17:29:30.147 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:29:30.880 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:29:30.880 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:29:30.669629Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:29:30.669629Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:29:30.669629Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:29:30.669629Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:29:30.669629Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:29:30.669629Z", "running": 0, "size": 1}}] 2022-02-21T17:29:40.906 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:29:41.627 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:29:41.628 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:29:41.439724Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:29:41.439724Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:29:41.439724Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:29:41.439724Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:29:41.439724Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:29:41.439724Z", "running": 0, "size": 1}}] 2022-02-21T17:29:51.656 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:29:52.384 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:29:52.385 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:29:52.198579Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:29:52.198579Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:29:52.198579Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:29:52.198579Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:29:52.198579Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:29:52.198579Z", "running": 0, "size": 1}}] 2022-02-21T17:30:02.406 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:30:03.458 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:30:03.459 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:30:02.929672Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:30:02.929672Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:30:02.929672Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:30:02.929672Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:30:02.929672Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:30:02.929672Z", "running": 0, "size": 1}}] 2022-02-21T17:30:13.485 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:30:14.221 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:30:14.221 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:30:14.016797Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:30:14.016797Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:30:14.016797Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:30:14.016797Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:30:14.016797Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:30:14.016797Z", "running": 0, "size": 1}}] 2022-02-21T17:30:24.250 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:30:24.996 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:30:24.996 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:30:24.778459Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:30:24.778459Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:30:24.778459Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:30:24.778459Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:30:24.778459Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:30:24.778459Z", "running": 0, "size": 1}}] 2022-02-21T17:30:35.020 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:30:35.706 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:30:35.706 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:30:35.510058Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:30:35.510058Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:30:35.510058Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:30:35.510058Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:30:35.510058Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:30:35.510058Z", "running": 0, "size": 1}}] 2022-02-21T17:30:45.732 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:30:46.632 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:30:46.632 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:30:46.410932Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:30:46.410932Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:30:46.410932Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:30:46.410932Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:30:46.410932Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:30:46.410932Z", "running": 0, "size": 1}}] 2022-02-21T17:30:56.659 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:30:57.387 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:30:57.388 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:30:57.186033Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:30:57.186033Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:30:57.186033Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:30:57.186033Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:30:57.186033Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:30:57.186033Z", "running": 0, "size": 1}}] 2022-02-21T17:31:07.417 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:31:08.168 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:31:08.168 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:31:07.951786Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:31:07.951786Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:31:07.951786Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:31:07.951786Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:31:07.951786Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:31:07.951786Z", "running": 0, "size": 1}}] 2022-02-21T17:31:18.196 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:31:18.963 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:31:18.963 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:31:18.761149Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:31:18.761149Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:31:18.761149Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:31:18.761149Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:31:18.761149Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:31:18.761149Z", "running": 0, "size": 1}}] 2022-02-21T17:31:28.993 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:31:29.745 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:31:29.746 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:31:29.538673Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:31:29.538673Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:31:29.538673Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:31:29.538673Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:31:29.538673Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:31:29.538673Z", "running": 0, "size": 1}}] 2022-02-21T17:31:39.768 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:31:40.483 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:31:40.484 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:31:40.266223Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:31:40.266223Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:31:40.266223Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:31:40.266223Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:31:40.266223Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:31:40.266223Z", "running": 0, "size": 1}}] 2022-02-21T17:31:50.513 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:31:51.270 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:31:51.271 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:31:51.049132Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:31:51.049132Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:31:51.049132Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:31:51.049132Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:31:51.049132Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:31:51.049132Z", "running": 0, "size": 1}}] 2022-02-21T17:32:01.297 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:32:02.031 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:32:02.031 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:32:01.836483Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:32:01.836483Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:32:01.836483Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:32:01.836483Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:32:01.836483Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:32:01.836483Z", "running": 0, "size": 1}}] 2022-02-21T17:32:12.057 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:32:12.821 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:32:12.822 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:32:12.583907Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:32:12.583907Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:32:12.583907Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:32:12.583907Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:32:12.583907Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:32:12.583907Z", "running": 0, "size": 1}}] 2022-02-21T17:32:22.856 DEBUG:teuthology.orchestra.run.smithi005:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-kqz7x -- ceph orch ls -f json 2022-02-21T17:32:23.656 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-02-21T17:32:23.657 INFO:teuthology.orchestra.run.smithi005.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:46.000000Z", "last_refresh": "2022-02-21T17:32:23.388969Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:10:34.000000Z", "last_refresh": "2022-02-21T17:32:23.388969Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:08:14.000000Z", "last_refresh": "2022-02-21T17:32:23.388969Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "f61e120f25aef5395aa2de70a53c352d63ab37e095939b85f5a0df1832b3075a", "container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "created": "2022-02-21T17:11:54.000000Z", "last_refresh": "2022-02-21T17:32:23.388969Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-02-21T17:32:23.388969Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:b05010b68efc838fdc35b41106e2ebdabe395a3e", "last_refresh": "2022-02-21T17:32:23.388969Z", "running": 0, "size": 1}}] 2022-02-21T17:32:23.682 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T17:32:23.683 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T17:32:23.683 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-02-21T17:32:23.684 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-02-21T17:32:23.699 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-02-21T17:32:23.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-02-21T17:32:23.734 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T17:32:23.735 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T17:32:23.748 DEBUG:teuthology.orchestra.remote:smithi005:rook/deploy/examples/operator.yaml is 20KB 2022-02-21T17:32:23.762 DEBUG:teuthology.orchestra.run.smithi005:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-02-21T17:32:23.832 INFO:teuthology.orchestra.run.smithi005.stdout:deployment.apps "rook-ceph-tools" deleted 2022-02-21T17:32:23.844 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b05010b68efc838fdc35b41106e2ebdabe395a3e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-02-21T17:32:23.844 DEBUG:teuthology.orchestra.run.smithi005:> kubectl delete -f cluster.yaml 2022-02-21T17:32:23.947 INFO:teuthology.orchestra.run.smithi005.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-02-21T17:32:23.955 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:23.935728 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-02-21T17:32:23.955 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:23.935882 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-02-21T17:32:23.955 INFO:tasks.rook.operator.smithi005.stdout:I0221 17:32:23.936034 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-02-21T17:32:23.959 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:23.936516 I | operator: successfully started the controller-runtime manager 2022-02-21T17:32:23.971 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:23.949662 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-02-21T17:32:23.971 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:23.949686 I | operator: watching all namespaces for Ceph CRs 2022-02-21T17:32:23.971 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:23.949722 I | operator: setting up schemes 2022-02-21T17:32:23.973 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:23.952355 I | operator: setting up the controller-runtime manager 2022-02-21T17:32:24.178 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.156624 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-02-21T17:32:24.181 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159511 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-02-21T17:32:24.181 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159554 I | ceph-cluster-controller: successfully started 2022-02-21T17:32:24.181 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159616 I | ceph-cluster-controller: enabling hotplug orchestration 2022-02-21T17:32:24.182 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159639 I | ceph-crashcollector-controller: successfully started 2022-02-21T17:32:24.182 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159688 I | ceph-block-pool-controller: successfully started 2022-02-21T17:32:24.182 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159722 I | ceph-object-store-user-controller: successfully started 2022-02-21T17:32:24.182 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159757 I | ceph-object-realm-controller: successfully started 2022-02-21T17:32:24.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159780 I | ceph-object-zonegroup-controller: successfully started 2022-02-21T17:32:24.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159799 I | ceph-object-zone-controller: successfully started 2022-02-21T17:32:24.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159934 I | ceph-object-controller: successfully started 2022-02-21T17:32:24.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.159977 I | ceph-file-controller: successfully started 2022-02-21T17:32:24.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160032 I | ceph-nfs-controller: successfully started 2022-02-21T17:32:24.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160074 I | ceph-rbd-mirror-controller: successfully started 2022-02-21T17:32:24.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160113 I | ceph-client-controller: successfully started 2022-02-21T17:32:24.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160140 I | ceph-filesystem-mirror-controller: successfully started 2022-02-21T17:32:24.184 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160165 I | operator: rook-ceph-operator-config-controller successfully started 2022-02-21T17:32:24.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160181 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-02-21T17:32:24.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160201 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-02-21T17:32:24.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160215 I | ceph-bucket-topic: successfully started 2022-02-21T17:32:24.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160228 I | ceph-bucket-notification: successfully started 2022-02-21T17:32:24.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160239 I | ceph-bucket-notification: successfully started 2022-02-21T17:32:24.186 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.160254 I | ceph-fs-subvolumegroup-controller: successfully started 2022-02-21T17:32:24.186 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.161656 I | operator: starting the controller-runtime manager 2022-02-21T17:32:24.286 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.267207 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-02-21T17:32:24.287 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.267289 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-02-21T17:32:24.287 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.267336 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-02-21T17:32:24.291 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.271432 I | operator: rook-ceph-operator-config-controller done reconciling 2022-02-21T17:32:24.292 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.272969 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:24.355 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.335031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:32:24.368 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.349134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:32:25.012 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:24.992819 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:24.982+0000 7f52dfe91340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:25.013 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:24.982+0000 7f52dfe91340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:25.013 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:24.982+0000 7f52dfe91340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:25.013 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:25.085 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:25.066190 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:25.970 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:25.950471 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:25.926+0000 7f1bc84a3340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:25.970 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:25.926+0000 7f1bc84a3340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:25.970 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:25.926+0000 7f1bc84a3340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:25.971 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:26.049 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:26.030208 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:26.703 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:26.684020 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:26.674+0000 7fc80c3e0340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:26.704 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:26.674+0000 7fc80c3e0340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:26.704 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:26.674+0000 7fc80c3e0340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:26.704 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:26.742 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:26.722683 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:27.451 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:27.431501 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:27.422+0000 7f0bbc184340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:27.452 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:27.422+0000 7f0bbc184340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:27.452 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:27.422+0000 7f0bbc184340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:27.452 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:27.527 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:27.508147 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:28.206 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:28.184047 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:28.174+0000 7fdae53cb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:28.206 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:28.174+0000 7fdae53cb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:28.206 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:28.174+0000 7fdae53cb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:28.206 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:28.291 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:28.272068 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:28.985 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:28.965337 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:28.954+0000 7fd4d4027340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:28.985 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:28.954+0000 7fd4d4027340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:28.985 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:28.954+0000 7fd4d4027340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:28.985 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:29.152 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:29.133081 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:29.828 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:29.808284 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:29.798+0000 7efd3b770340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:29.828 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:29.798+0000 7efd3b770340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:29.828 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:29.798+0000 7efd3b770340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:29.829 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:30.154 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:30.134748 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:30.824 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:30.804361 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:30.790+0000 7fac9f05f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:30.824 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:30.790+0000 7fac9f05f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:30.824 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:30.790+0000 7fac9f05f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:30.825 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:31.469 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:31.450542 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:32.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:32.137314 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:32.126+0000 7fdf26d77340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:32.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:32.126+0000 7fdf26d77340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:32.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:32.126+0000 7fdf26d77340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:32.158 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:33.451 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:33.432065 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:34.160 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:34.141065 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:34.130+0000 7fc43e024340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:34.404 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:34.130+0000 7fc43e024340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:34.404 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:34.130+0000 7fc43e024340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:34.404 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:34.470 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:34.451382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:32:34.483 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:34.463945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:32:36.729 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:36.710238 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:37.402 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:37.382790 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:37.374+0000 7fddf2f02340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:37.402 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:37.374+0000 7fddf2f02340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:37.403 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:37.374+0000 7fddf2f02340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:37.403 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:42.531 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:42.512167 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:43.214 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:43.194978 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:43.186+0000 7f88309a0340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:43.215 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:43.186+0000 7f88309a0340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:43.215 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:43.186+0000 7f88309a0340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:43.215 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:44.549 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:44.529850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:32:44.563 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:44.544641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:32:53.463 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:53.444587 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:32:54.104 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:54.084323 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:32:54.074+0000 7f3123eca340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:32:54.104 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:54.074+0000 7f3123eca340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:32:54.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:32:54.074+0000 7f3123eca340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:32:54.105 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:32:54.681 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:54.661671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:32:54.694 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:32:54.672276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:04.812 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:04.792499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:04.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:04.806146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:14.591 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:14.572463 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:33:14.893 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:14.872898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:14.903 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:14.883940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:15.248 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:15.228517 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:33:15.217+0000 7f097fe0f340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:33:15.248 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:33:15.217+0000 7f097fe0f340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:33:15.248 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:33:15.217+0000 7f097fe0f340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:33:15.249 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:33:25.019 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:25.000164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:25.034 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:25.015161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:35.131 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:35.112210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:35.146 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:35.127302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:45.213 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:45.194111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:45.225 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:45.206345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:55.298 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:55.278801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:55.310 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:55.290956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:33:56.220 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:56.200933 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:33:56.876 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:33:56.856639 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:33:56.845+0000 7f8970772340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:33:56.876 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:33:56.845+0000 7f8970772340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:33:56.877 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:33:56.845+0000 7f8970772340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:33:56.877 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:34:05.431 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:05.412345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:05.446 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:05.427225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:15.574 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:15.555220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:15.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:15.569288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:25.712 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:25.693106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:25.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:25.703756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:35.830 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:35.810680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:35.844 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:35.825271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:45.957 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:45.937912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:45.971 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:45.952600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:56.065 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:56.046344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:34:56.081 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:34:56.062150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:06.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:06.137767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:06.169 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:06.150713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:16.262 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:16.242843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:16.277 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:16.257974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:18.804 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:18.785196 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:35:19.459 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:19.440239 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:35:19.427+0000 7f4dc2477340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:35:19.460 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:35:19.427+0000 7f4dc2477340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:35:19.460 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:35:19.427+0000 7f4dc2477340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:35:19.460 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:35:26.347 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:26.327649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:26.361 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:26.342842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:36.439 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:36.420149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:36.453 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:36.433949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:46.575 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:46.556385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:46.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:46.568915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:56.676 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:56.657056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:35:56.689 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:35:56.670820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:06.803 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:06.783602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:06.819 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:06.800019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:16.885 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:16.866465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:16.900 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:16.881212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:26.961 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:26.941621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:27.008 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:26.988927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:37.107 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:37.088379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:37.118 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:37.099581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:47.197 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:47.178547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:47.210 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:47.190921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:57.277 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:57.258257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:36:57.292 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:36:57.273357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:07.396 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:07.375798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:07.408 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:07.389581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:17.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:17.477339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:17.508 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:17.489202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:27.636 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:27.617169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:27.648 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:27.629363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:37.762 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:37.742312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:37.773 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:37.754613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:47.862 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:47.842465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:47.905 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:47.886135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:57.996 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:57.977387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:37:58.009 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:37:57.989768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:03.308 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:03.289512 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:38:03.992 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:03.973071 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:38:03.964+0000 7f7935e36340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:38:03.993 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:38:03.964+0000 7f7935e36340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:38:03.993 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:38:03.964+0000 7f7935e36340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:38:03.993 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:38:08.173 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:08.153317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:08.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:08.166048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:18.256 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:18.236566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:18.295 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:18.275920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:28.403 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:28.383879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:28.416 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:28.396752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:38.519 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:38.500086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:38.530 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:38.510871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:48.598 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:48.578896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:48.611 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:48.592455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:58.682 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:58.663118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:38:58.694 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:38:58.674971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:08.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:08.777562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:08.811 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:08.791430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:18.944 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:18.925050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:18.956 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:18.936640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:29.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:29.005489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:29.082 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:29.063368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:39.198 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:39.173642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:39.206 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:39.186911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:49.273 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:49.253667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:49.287 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:49.267829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:59.378 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:59.358469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:39:59.391 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:39:59.371963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:09.489 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:09.469571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:09.510 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:09.490528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:19.611 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:19.591561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:19.626 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:19.606904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:29.691 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:29.671604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:29.703 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:29.683917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:39.788 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:39.769388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:39.800 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:39.781299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:49.874 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:49.854612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:40:49.887 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:40:49.867938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:00.021 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:00.001413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:00.035 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:00.016188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:10.102 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:10.082643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:10.114 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:10.094257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:20.226 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:20.206489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:20.241 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:20.222333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:30.362 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:30.343088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:30.373 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:30.354213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:40.492 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:40.472659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:40.506 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:40.486842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:50.608 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:50.588303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:41:50.620 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:41:50.600687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:00.730 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:00.711148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:00.745 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:00.726515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:10.864 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:10.845193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:10.879 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:10.859552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:20.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:20.966069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:20.999 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:20.980349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:31.108 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:31.088750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:31.122 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:31.103149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:41.195 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:41.175660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:41.209 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:41.189865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:51.286 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:51.266662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:42:51.299 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:42:51.279851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:01.436 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:01.417008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:01.450 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:01.431056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:11.548 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:11.528296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:11.558 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:11.539261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:21.666 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:21.647048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:21.680 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:21.660641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:31.688 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:31.668236 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:43:31.757 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:31.737597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:31.770 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:31.751424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:32.384 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:32.364120 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:43:32.353+0000 7ff8459e5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:43:32.384 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:43:32.353+0000 7ff8459e5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:43:32.384 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:43:32.353+0000 7ff8459e5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:43:32.384 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:43:41.876 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:41.856800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:41.889 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:41.870147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:51.998 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:51.979126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:43:52.011 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:43:51.992581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:02.122 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:02.104189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:02.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:02.137974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:12.269 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:12.251438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:12.285 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:12.267455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:22.410 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:22.392171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:22.424 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:22.406678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:32.500 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:32.483360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:32.515 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:32.497925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:42.625 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:42.607592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:42.676 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:42.659760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:52.760 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:52.742894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:44:52.774 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:44:52.757520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:02.882 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:02.866157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:02.893 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:02.877189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:12.961 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:12.945045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:12.975 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:12.959620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:23.109 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:23.093606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:23.125 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:23.109535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:33.235 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:33.219797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:33.248 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:33.233561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:43.356 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:43.341056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:43.368 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:43.353060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:53.484 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:53.469879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:45:53.565 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:45:53.550335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:03.634 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:03.619373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:03.646 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:03.631918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:13.707 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:13.693098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:13.775 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:13.760565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:23.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:23.827723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:23.856 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:23.842642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:33.925 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:33.911635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:33.941 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:33.928042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:44.012 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:43.998811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:44.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:44.012252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:54.097 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:54.083969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:46:54.110 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:46:54.097299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:04.236 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:04.223750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:04.250 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:04.237099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:14.323 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:14.310444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:14.334 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:14.322146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:24.436 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:24.423704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:24.462 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:24.450344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:34.572 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:34.559518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:34.585 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:34.573213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:44.698 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:44.686340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:44.711 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:44.699489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:54.846 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:54.834756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:47:54.858 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:47:54.847124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:04.969 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:04.957847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:04.983 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:04.971744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:15.082 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:15.070831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:15.095 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:15.084645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:25.199 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:25.187736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:25.211 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:25.200823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:35.333 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:35.321924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:35.345 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:35.334023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:45.410 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:45.399404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:45.424 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:45.413655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:55.536 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:55.525617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:48:55.548 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:48:55.537966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:05.721 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:05.710529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:05.738 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:05.723858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:15.854 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:15.844003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:15.866 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:15.856201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:25.929 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:25.919654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:25.941 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:25.931577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:36.006 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:35.995979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:36.017 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:36.007629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:46.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:46.088395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:46.137 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:46.127526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:56.203 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:56.193426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:49:56.261 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:49:56.249917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:06.339 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:06.329619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:06.374 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:06.364925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:16.498 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:16.489099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:16.511 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:16.502421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:26.620 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:26.610619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:26.653 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:26.644138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:36.742 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:36.732871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:36.754 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:36.745862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:46.820 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:46.811509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:46.893 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:46.884438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:57.000 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:56.992071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:50:57.014 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:50:57.006377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:07.124 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:07.115694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:07.136 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:07.128380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:17.314 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:17.305703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:17.326 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:17.318028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:27.389 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:27.377631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:27.400 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:27.391904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:37.501 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:37.492854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:37.538 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:37.528082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:47.604 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:47.596170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:47.665 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:47.657608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:57.763 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:57.755605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:51:57.774 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:51:57.766864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:07.871 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:07.863139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:07.883 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:07.876123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:17.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:17.971721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:17.991 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:17.983847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:28.090 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:28.082638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:28.103 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:28.096437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:38.198 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:38.190734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:38.212 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:38.205000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:48.297 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:48.290213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:48.358 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:48.350891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:58.466 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:58.459502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:52:58.483 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:52:58.476818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:08.556 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:08.549691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:08.571 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:08.564716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:18.644 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:18.636994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:18.702 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:18.694962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:28.762 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:28.755427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:28.776 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:28.769587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:38.850 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:38.843408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:38.907 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:38.901040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:48.976 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:48.969588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:48.991 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:48.984246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:59.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:59.091864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:53:59.126 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:53:59.120430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:09.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:09.251181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:09.268 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:09.262633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:19.339 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:19.333038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:19.353 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:19.347566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:27.739 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:27.733529 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T17:54:28.397 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:28.390880 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T17:54:28.382+0000 7f1153c8e340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T17:54:28.398 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:54:28.382+0000 7f1153c8e340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T17:54:28.398 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T17:54:28.382+0000 7f1153c8e340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T17:54:28.398 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T17:54:29.429 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:29.423647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:29.442 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:29.436303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:39.529 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:39.522974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:39.541 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:39.535500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:49.605 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:49.598951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:49.619 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:49.613504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:59.693 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:59.687124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:54:59.742 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:54:59.736389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:09.854 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:09.848301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:09.868 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:09.862017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:19.945 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:19.939363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:19.968 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:19.962654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:30.042 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:30.036850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:30.055 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:30.050281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:40.148 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:40.142757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:40.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:40.153764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:50.229 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:50.224264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:55:50.245 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:55:50.239656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:00.315 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:00.310270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:00.369 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:00.364420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:10.472 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:10.467382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:10.484 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:10.479127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:20.590 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:20.585231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:20.609 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:20.603912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:30.682 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:30.676998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:30.694 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:30.689996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:40.758 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:40.753830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:40.769 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:40.765165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:50.919 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:50.914957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:56:50.932 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:56:50.927800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:01.039 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:01.034789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:01.056 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:01.051889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:11.162 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:11.157350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:11.173 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:11.169668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:21.272 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:21.267448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:21.285 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:21.281162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:31.437 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:31.432646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:31.449 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:31.444768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:41.524 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:41.519688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:41.537 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:41.533883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:51.636 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:51.632044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:57:51.649 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:57:51.645110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:01.708 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:01.703805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:01.778 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:01.774527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:11.883 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:11.879184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:11.896 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:11.892364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:22.041 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:22.037520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:22.053 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:22.048991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:32.154 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:32.150387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:32.167 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:32.163567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:42.261 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:42.257890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:42.274 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:42.270652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:52.375 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:52.371634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:58:52.388 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:58:52.384779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:02.449 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:02.445816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:02.461 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:02.458212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:12.564 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:12.560508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:12.577 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:12.574261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:22.683 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:22.679716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:22.698 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:22.694925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:32.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:32.821531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:32.837 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:32.833678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:42.897 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:42.890740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:42.908 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:42.905910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:52.967 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:52.964065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T17:59:52.980 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 17:59:52.976986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:03.095 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:03.092165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:03.273 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:03.104820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:13.188 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:13.185067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:13.201 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:13.197917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:23.312 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:23.309302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:23.327 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:23.323887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:33.430 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:33.426836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:33.443 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:33.440165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:43.500 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:43.497882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:43.512 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:43.509848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:53.580 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:53.577747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:00:53.595 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:00:53.592660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:03.729 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:03.726802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:03.740 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:03.738038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:13.856 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:13.853118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:13.869 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:13.866063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:23.937 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:23.934631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:23.949 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:23.946624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:34.058 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:34.055171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:34.073 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:34.070168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:44.149 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:44.141914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:44.158 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:44.155645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:54.288 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:54.284858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:01:54.299 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:01:54.297067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:04.366 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:04.363445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:04.378 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:04.375840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:14.472 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:14.470082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:14.486 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:14.483912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:24.566 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:24.563729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:24.624 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:24.621676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:34.691 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:34.688603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:34.705 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:34.703302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:44.812 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:44.809360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:44.824 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:44.822602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:54.932 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:54.929494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:02:54.949 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:02:54.943976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:05.059 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:05.057457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:05.075 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:05.073002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:15.224 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:15.222257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:15.236 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:15.234564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:25.341 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:25.339284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:25.352 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:25.350790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:35.466 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:35.463802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:35.477 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:35.475264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:45.546 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:45.544104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:45.594 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:45.591949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:55.705 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:55.703595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:03:55.717 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:03:55.715713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:05.910 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:05.908347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:05.919 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:05.917574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:16.021 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:16.019602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:16.037 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:16.034615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:26.147 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:26.145560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:26.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:26.157822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:36.231 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:36.229866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:36.245 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:36.242932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:46.321 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:46.319459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:46.334 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:46.332737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:56.409 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:56.407642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:04:56.448 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:04:56.441575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:06.603 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:06.601805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:06.615 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:06.613440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:16.681 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:16.680139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:16.695 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:16.694099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:26.807 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:26.805444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:26.818 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:26.817137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:36.930 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:36.927837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:36.941 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:36.938337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:47.046 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:47.045178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:47.059 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:47.057897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:57.168 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:57.166703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:05:57.423 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:05:57.179649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:07.298 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:07.296150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:07.308 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:07.307446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:17.378 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:17.377566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:17.390 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:17.389722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:27.486 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:27.484714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:27.584 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:27.500855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:37.586 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:37.585645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:37.600 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:37.599843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:47.674 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:47.673156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:47.689 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:47.688724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:57.783 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:57.782623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:06:57.796 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:06:57.795025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:07.881 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:07.880618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:07.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:07.947147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:18.019 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:18.018595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:18.032 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:18.032272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:28.146 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:28.145262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:28.158 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:28.157512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:38.285 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:38.284299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:38.296 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:38.295471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:48.366 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:48.364059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:48.379 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:48.378378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:58.485 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:58.484154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:07:58.498 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:07:58.496907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:08.624 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:08.623574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:08.638 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:08.637195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:18.717 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:18.716062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:18.777 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:18.776562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:28.845 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:28.844176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:28.856 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:28.856129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:38.959 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:38.958329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:38.968 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:38.968261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:49.051 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:49.049801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:49.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:49.063752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:59.209 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:59.208223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:08:59.218 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:08:59.218060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:09.355 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:09.354432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:09.367 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:09.366849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:19.505 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:19.502655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:19.513 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:19.513173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:29.663 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:29.662859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:29.676 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:29.675842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:39.787 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:39.786881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:39.800 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:39.800034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:49.958 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:49.957648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:09:49.991 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:09:49.991224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:00.092 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:00.091821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:00.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:00.105549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:10.189 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:10.188982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:10.203 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:10.203188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:20.306 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:20.305601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:20.319 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:20.318422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:30.391 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:30.391486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:30.403 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:30.403483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:40.467 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:40.467101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:40.479 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:40.479432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:50.543 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:50.542555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:10:50.555 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:10:50.554841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:00.626 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:00.626322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:00.640 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:00.639974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:08.403 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:08.402543 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T18:11:09.062 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:09.061512 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:11:09.048+0000 7fb9ce562340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:11:09.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T18:11:09.048+0000 7fb9ce562340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:11:09.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T18:11:09.048+0000 7fb9ce562340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:11:09.063 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:11:10.748 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:10.747846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:10.761 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:10.761346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:20.875 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:20.873796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:20.883 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:20.883238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:30.991 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:30.991396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:31.003 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:31.003643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:41.066 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:41.066232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:41.150 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:41.149865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:51.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:51.256878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:11:51.268 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:11:51.268864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:01.418 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:01.418339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:01.459 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:01.459526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:11.560 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:11.560063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:11.573 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:11.572712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:21.626 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:21.624233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:21.695 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:21.695636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:31.762 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:31.761885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:31.774 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:31.774293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:41.840 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:41.840407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:41.854 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:41.854742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:51.935 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:51.934905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:12:51.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:12:51.948320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:02.015 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:02.015170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:02.028 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:02.028845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:12.092 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:12.092498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:12.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:12.104845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:22.224 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:22.224363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:22.237 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:22.237878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:32.324 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:32.324132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:32.338 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:32.337666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:42.409 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:42.409054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:42.422 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:42.422003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:52.509 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:52.509909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:13:52.527 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:13:52.526891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:02.629 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:02.629355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:02.642 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:02.642179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:12.765 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:12.765623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:12.779 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:12.780061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:22.838 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:22.838475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:22.851 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:22.851824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:32.951 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:32.951425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:32.989 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:32.989074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:43.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:43.063758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:43.078 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:43.078169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:53.167 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:53.167211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:14:53.180 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:14:53.181038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:03.260 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:03.260667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:03.272 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:03.273053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:13.374 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:13.374211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:13.386 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:13.387211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:23.461 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:23.461644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:23.535 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:23.535239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:33.617 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:33.617126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:33.627 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:33.628371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:43.742 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:43.742845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:43.755 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:43.756184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:53.832 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:53.832214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:15:53.844 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:15:53.844465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:03.936 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:03.936465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:03.947 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:03.948385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:14.061 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:14.061310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:14.073 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:14.074008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:24.182 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:24.182518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:24.191 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:24.191773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:34.286 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:34.286024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:34.298 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:34.298880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:44.387 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:44.387670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:44.415 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:44.416097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:54.555 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:54.555702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:16:54.568 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:16:54.569073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:04.659 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:04.660089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:04.672 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:04.672536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:14.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:14.797537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:14.810 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:14.810823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:24.918 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:24.919374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:24.930 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:24.931425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:35.033 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:35.033633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:35.048 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:35.048062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:45.113 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:45.114436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:45.124 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:45.125645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:55.241 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:55.242265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:17:55.253 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:17:55.254821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:05.420 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:05.421284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:05.433 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:05.435248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:15.559 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:15.560701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:15.574 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:15.575686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:25.683 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:25.685758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:25.699 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:25.702136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:35.814 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:35.816562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:35.828 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:35.830755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:45.895 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:45.898504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:45.909 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:45.912304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:55.990 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:55.992660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:18:56.002 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:18:56.005159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:06.111 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:06.114825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:06.124 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:06.127457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:16.193 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:16.196802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:16.213 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:16.210663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:26.278 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:26.281925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:26.345 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:26.348390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:36.451 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:36.455289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:36.464 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:36.469086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:46.594 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:46.597803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:46.609 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:46.613832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:56.698 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:56.703033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:19:56.710 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:19:56.715682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:06.817 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:06.821628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:06.829 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:06.834222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:16.971 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:16.976304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:16.984 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:16.990015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:27.067 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:27.072064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:27.077 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:27.083344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:37.166 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:37.171921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:37.179 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:37.184960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:47.329 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:47.335126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:47.345 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:47.351259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:57.420 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:57.426605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:20:57.435 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:20:57.438960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:07.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:07.502388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:07.559 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:07.565660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:17.629 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:17.636293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:17.641 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:17.648343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:27.736 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:27.742551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:27.748 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:27.755008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:37.819 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:37.826289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:37.831 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:37.838352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:47.904 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:47.911413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:47.922 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:47.925157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:57.984 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:57.991258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:21:57.995 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:21:58.002848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:08.086 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:08.093403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:08.097 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:08.105267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:18.158 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:18.166301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:18.169 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:18.177779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:28.294 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:28.302222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:28.305 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:28.313568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:38.414 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:38.422786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:38.427 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:38.436025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:48.527 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:48.535922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:48.540 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:48.548987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:58.609 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:58.618059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:22:58.623 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:22:58.632331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:08.689 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:08.698010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:08.704 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:08.713481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:18.774 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:18.783591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:18.822 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:18.831507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:28.901 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:28.910289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:28.957 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:28.966837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:39.137 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:39.147131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:39.227 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:39.234667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:49.350 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:49.359362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:49.364 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:49.374096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:59.498 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:59.507867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:23:59.511 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:23:59.521148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:09.578 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:09.588461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:09.679 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:09.687601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:19.783 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:19.793847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:19.795 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:19.805569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:29.941 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:29.951231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:29.953 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:29.963795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:40.024 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:40.035285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:40.037 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:40.048046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:50.128 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:50.139379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:24:50.140 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:24:50.151182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:00.212 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:00.223383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:00.259 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:00.270416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:10.371 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:10.382385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:10.386 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:10.397810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:20.461 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:20.469061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:20.472 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:20.483040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:30.539 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:30.550625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:30.552 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:30.563643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:40.629 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:40.639830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:40.642 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:40.653715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:50.728 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:50.739461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:25:50.742 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:25:50.754041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:00.833 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:00.845204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:00.847 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:00.859083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:10.916 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:10.928321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:10.932 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:10.944221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:21.007 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:21.019458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:21.020 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:21.032654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:31.112 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:31.124411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:31.124 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:31.137453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:41.233 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:41.245998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:41.246 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:41.258606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:51.317 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:51.329950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:26:51.329 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:26:51.342768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:01.401 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:01.414195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:01.416 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:01.429007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:11.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:11.509278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:11.508 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:11.522004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:21.576 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:21.589799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:21.590 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:21.603348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:31.663 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:31.676353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:31.676 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:31.689422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:41.770 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:41.783422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:41.786 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:41.799197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:49.057 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:49.070946 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T18:27:49.739 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:49.753059 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:27:49.741+0000 7f5b7fc1b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:27:49.740 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T18:27:49.741+0000 7f5b7fc1b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:27:49.740 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T18:27:49.741+0000 7f5b7fc1b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:27:49.740 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:27:51.859 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:51.873216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:27:51.873 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:27:51.887004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:01.992 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:02.006395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:02.004 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:02.018335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:12.068 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:12.082274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:12.082 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:12.096234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:22.147 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:22.161487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:22.160 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:22.174194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:32.261 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:32.275929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:32.274 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:32.288954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:42.341 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:42.355400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:42.354 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:42.369172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:52.476 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:52.490625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:28:52.489 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:28:52.504401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:02.595 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:02.609880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:02.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:02.622215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:12.706 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:12.720501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:12.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:12.733579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:22.826 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:22.840591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:22.840 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:22.855909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:32.911 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:32.925917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:32.924 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:32.939579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:42.996 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:43.011837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:43.030 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:43.045699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:53.104 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:53.119662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:29:53.115 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:29:53.131039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:03.318 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:03.333781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:03.375 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:03.390900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:13.441 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:13.456633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:13.463 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:13.478910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:23.599 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:23.614901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:23.614 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:23.630098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:33.678 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:33.693962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:33.692 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:33.708006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:43.799 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:43.814646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:43.812 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:43.828320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:53.904 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:53.920571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:30:53.919 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:30:53.935149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:04.001 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:04.017166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:04.015 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:04.031199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:14.110 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:14.126179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:14.124 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:14.140774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:24.200 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:24.216399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:24.213 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:24.229773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:34.274 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:34.290537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:34.347 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:34.363666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:44.456 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:44.473115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:44.469 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:44.486412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:54.541 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:54.557858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:31:54.554 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:31:54.571142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:04.630 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:04.647021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:04.643 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:04.660098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:14.773 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:14.789849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:14.784 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:14.801698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:24.884 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:24.900778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:24.896 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:24.913665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:34.967 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:34.984181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:34.980 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:34.997885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:45.056 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:45.073242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:45.068 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:45.085849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:55.132 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:55.149598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:32:55.147 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:32:55.164141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:05.230 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:05.247148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:05.246 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:05.263690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:15.340 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:15.357973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:15.353 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:15.370205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:25.426 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:25.443535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:25.494 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:25.512159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:35.571 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:35.589264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:35.638 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:35.656411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:45.734 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:45.752310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:45.751 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:45.768984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:55.871 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:55.889142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:33:55.883 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:33:55.901474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:05.995 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:06.012958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:06.026 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:06.044568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:16.135 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:16.153081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:16.149 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:16.167624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:26.265 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:26.283558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:26.277 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:26.296116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:36.373 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:36.391739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:36.386 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:36.404711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:46.498 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:46.516094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:46.511 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:46.529381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:56.612 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:56.631282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:34:56.624 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:34:56.642703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:06.698 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:06.716552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:06.710 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:06.729319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:16.782 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:16.800746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:16.793 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:16.811875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:26.873 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:26.892448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:26.884 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:26.902962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:36.951 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:36.969751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:36.962 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:36.980915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:47.058 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:47.076886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:47.119 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:47.138235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:57.188 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:57.207687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:35:57.202 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:35:57.221249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:07.312 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:07.331034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:07.325 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:07.344330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:17.404 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:17.423622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:17.416 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:17.435836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:27.562 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:27.581252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:27.572 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:27.591878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:37.676 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:37.695142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:37.743 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:37.762360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:47.868 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:47.887815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:47.880 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:47.899933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:57.945 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:57.964749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:36:57.958 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:36:57.977873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:08.052 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:08.071921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:08.066 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:08.086032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:18.168 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:18.187742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:18.178 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:18.198167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:28.279 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:28.299242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:28.292 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:28.312684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:38.405 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:38.424941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:38.417 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:38.437893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:48.524 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:48.543901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:48.537 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:48.557034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:58.625 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:58.644314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:37:59.067 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:37:58.656326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:08.757 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:08.777763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:08.773 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:08.793023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:18.896 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:18.915965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:18.914 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:18.934470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:29.015 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:29.031446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:29.026 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:29.046252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:39.100 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:39.118367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:39.118 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:39.131565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:49.275 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:49.295895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:49.291 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:49.308245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:59.377 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:59.398025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:38:59.389 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:38:59.409729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:09.473 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:09.493509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:09.545 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:09.566044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:19.613 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:19.633022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:19.644 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:19.664733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:29.743 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:29.763543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:29.755 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:29.776210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:39.863 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:39.884179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:39.879 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:39.900080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:49.992 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:50.012691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:39:50.005 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:39:50.026229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:00.074 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:00.095400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:00.085 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:00.106523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:10.164 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:10.185465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:10.235 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:10.256288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:20.378 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:20.399611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:20.389 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:20.411362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:30.495 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:30.516207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:30.508 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:30.529684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:40.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:40.627991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:40.621 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:40.642502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:50.691 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:50.712312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:40:50.705 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:40:50.726787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:00.845 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:00.866769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:00.858 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:00.879758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:10.928 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:10.949786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:10.943 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:10.965128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:21.041 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:21.062625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:21.052 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:21.074101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:31.142 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:31.164000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:31.154 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:31.176414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:41.258 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:41.280029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:41.272 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:41.294198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:51.342 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:51.363719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:41:51.358 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:41:51.380195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:01.429 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:01.451082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:01.442 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:01.464727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:11.528 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:11.549902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:11.539 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:11.561824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:21.608 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:21.625536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:21.682 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:21.704557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:31.764 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:31.786779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:31.781 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:31.803471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:41.866 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:41.888015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:41.879 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:41.901555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:52.003 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:52.025089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:42:52.012 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:42:52.035130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:02.122 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:02.144138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:02.136 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:02.158374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:12.246 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:12.268800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:12.260 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:12.282736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:22.335 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:22.357897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:22.349 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:22.371479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:32.469 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:32.491744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:32.482 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:32.504461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:42.550 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:42.572980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:42.564 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:42.587350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:52.652 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:52.674359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:43:52.664 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:43:52.687127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:02.757 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:02.779834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:02.777 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:02.799924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:12.889 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:12.911587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:12.904 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:12.926210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:22.974 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:22.996249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:22.988 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:23.011195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:29.769 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:29.792187 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T18:44:30.476 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:30.498629 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T18:44:30.483+0000 7fe19942c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T18:44:30.476 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T18:44:30.483+0000 7fe19942c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T18:44:30.476 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T18:44:30.483+0000 7fe19942c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T18:44:30.477 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T18:44:33.094 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:33.116920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:33.107 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:33.130259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:43.220 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:43.243212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:43.233 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:43.256415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:53.338 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:53.360800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:44:53.348 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:44:53.371631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:03.422 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:03.445572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:03.435 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:03.458744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:13.558 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:13.580444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:13.574 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:13.597156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:23.649 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:23.672471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:23.667 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:23.690116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:33.747 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:33.770502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:33.761 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:33.784673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:43.829 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:43.853040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:43.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:43.865572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:53.917 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:53.938444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:45:53.927 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:45:53.951390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:04.029 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:04.052566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:04.043 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:04.066804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:14.122 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:14.144888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:14.134 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:14.157976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:24.241 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:24.264377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:24.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:24.281168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:34.328 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:34.351625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:34.404 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:34.427719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:44.510 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:44.534071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:44.523 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:44.546997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:54.633 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:54.657099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:46:54.653 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:46:54.673716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:04.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:04.749988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:04.784 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:04.808539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:14.854 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:14.877848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:14.868 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:14.891907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:24.957 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:24.980914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:24.976 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:25.000131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:35.112 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:35.136361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:35.127 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:35.151252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:45.202 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:45.226231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:45.216 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:45.240136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:55.286 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:55.310200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:47:55.343 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:47:55.367631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:05.438 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:05.461257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:05.454 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:05.478049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:15.574 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:15.598171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:15.592 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:15.616400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:25.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:25.742828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:25.731 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:25.756021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:35.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:35.865625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:35.854 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:35.879125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:45.953 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:45.977069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:45.968 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:45.992369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:56.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:56.087039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:48:56.077 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:48:56.101554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:06.223 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:06.247082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:06.237 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:06.261404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:16.321 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:16.343187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:16.331 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:16.355575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:26.439 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:26.463828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:26.451 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:26.475677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:36.551 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:36.575242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:36.565 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:36.590509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:46.699 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:46.723390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:46.713 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:46.738170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:56.778 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:56.803224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:49:56.800 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:49:56.821738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:06.943 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:06.968096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:06.956 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:06.981173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:17.041 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:17.066571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:17.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:17.079402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:27.122 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:27.147584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:27.133 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:27.158683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:37.225 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:37.249904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:37.239 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:37.264315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:47.323 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:47.347906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:47.335 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:47.360194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:57.414 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:57.439629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:50:57.437 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:50:57.462451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:07.521 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:07.546352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:07.541 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:07.563102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:17.631 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:17.656636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:17.642 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:17.667895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:27.707 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:27.732639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:27.721 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:27.746517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:37.826 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:37.851520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:37.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:37.866777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:47.937 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:47.961718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:47.947 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:47.972699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:58.048 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:58.073709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:51:58.062 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:51:58.087904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:08.171 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:08.197030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:08.186 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:08.211556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:18.274 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:18.299501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:18.285 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:18.310919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:28.359 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:28.384873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:28.426 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:28.451833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:38.500 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:38.525613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:38.513 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:38.538706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:48.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:48.613475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:48.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:48.627588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:58.705 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:58.730024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:52:58.716 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:52:58.741792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:08.778 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:08.803521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:08.796 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:08.821795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:18.922 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:18.947737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:18.938 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:18.961947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:29.052 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:29.077785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:29.065 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:29.091515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:39.196 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:39.222073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:39.208 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:39.234217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:49.315 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:49.341030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:49.376 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:49.401994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:59.459 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:59.480900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:53:59.471 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:53:59.494623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:09.565 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:09.591532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:09.580 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:09.606549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:19.641 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:19.667184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:19.654 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:19.680395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:29.719 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:29.745290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:29.733 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:29.758935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:39.799 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:39.825860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:39.811 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:39.838192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:49.885 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:49.911295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:49.904 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:54:49.925558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:59.977 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:00.003823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:54:59.989 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:00.015930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:10.102 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:10.128136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:10.114 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:10.140132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:20.229 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:20.254927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:20.243 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:20.270000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:30.309 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:30.335750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:30.325 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:30.351440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:40.419 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:40.445460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:40.431 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:40.457865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:50.518 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:50.545279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:55:50.534 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:55:50.561531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:00.635 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:00.661584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:00.647 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:00.674374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:10.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:10.753848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:10.740 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:10.767234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:20.915 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:20.942399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:20.930 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:20.957108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:31.000 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:31.027085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:31.012 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:31.039303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:41.118 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:41.141967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:41.126 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:41.153003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:51.250 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:51.277340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:56:51.280 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:56:51.288506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:01.329 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:01.356073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:01.374 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:01.401007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:11.455 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:11.482077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:11.471 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:11.498230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:21.576 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:21.603347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:21.590 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:21.617406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:31.690 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:31.717191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:31.703 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:31.730947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:41.784 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:41.811096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:41.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:41.824080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:51.871 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:51.898852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:57:51.887 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:57:51.914902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:01.964 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:01.991858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:01.978 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:02.005282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:12.041 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:12.068989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:12.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:12.082146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:22.148 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:22.175879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:22.162 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:22.190184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:32.292 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:32.319798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:32.308 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:32.336340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:42.423 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:42.451081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:42.434 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:42.461933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:52.490 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:52.517353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:58:52.502 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:58:52.529712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:02.599 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:02.625662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:02.610 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:02.638232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:12.679 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:12.706596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:12.694 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:12.721840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:22.784 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:22.812133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:22.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:22.825541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:32.878 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:32.905669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:32.895 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:32.922240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:42.960 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:42.987530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:42.973 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:43.001553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:53.069 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:53.096974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T18:59:53.081 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 18:59:53.108942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:03.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:03.183524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:03.171 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:03.198838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:13.237 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:13.264485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:13.280 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:13.307492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:23.347 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:23.374843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:23.377 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:23.405391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:33.475 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:33.503689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:33.488 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:33.516703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:43.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:43.634997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:43.619 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:43.647138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:53.712 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:53.740438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:00:53.726 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:00:53.754572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:03.882 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:03.910668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:03.896 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:03.925047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:10.477 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:10.504952 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T19:01:11.202 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:11.230013 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:01:11.220+0000 7fc8344e5340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:01:11.202 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T19:01:11.220+0000 7fc8344e5340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:01:11.203 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T19:01:11.220+0000 7fc8344e5340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:01:11.203 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:01:13.974 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:14.002231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:14.045 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:14.073040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:24.115 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:24.143304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:24.130 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:24.158748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:34.270 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:34.298418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:34.304 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:34.332394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:44.413 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:44.441042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:44.427 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:44.456106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:54.502 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:54.530699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:01:54.512 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:01:54.541048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:04.645 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:04.673395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:04.659 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:04.688111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:14.757 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:14.785246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:14.771 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:14.800366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:24.872 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:24.900293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:24.884 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:24.912705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:34.990 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:35.019154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:35.003 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:35.031652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:45.079 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:45.108144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:45.095 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:45.123171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:55.209 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:55.237553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:02:55.221 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:02:55.250620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:05.415 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:05.443837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:05.428 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:05.456895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:15.595 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:15.623690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:15.607 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:15.635881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:25.745 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:25.773653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:25.759 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:25.787866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:35.882 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:35.910812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:35.895 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:35.924099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:46.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:46.053579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:46.037 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:46.066644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:56.207 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:56.235784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:03:56.220 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:03:56.249768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:06.321 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:06.350082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:06.333 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:06.362134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:16.405 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:16.434436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:16.419 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:16.448475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:26.543 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:26.572211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:26.556 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:26.585268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:36.670 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:36.699179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:36.685 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:36.714595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:46.810 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:46.839278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:46.824 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:46.853732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:56.924 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:56.953237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:04:56.977 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:04:57.006165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:07.041 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:07.070530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:07.086 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:07.115418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:17.201 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:17.230057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:17.212 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:17.241654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:27.288 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:27.317525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:27.300 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:27.329941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:37.370 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:37.399738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:37.385 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:37.414537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:47.460 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:47.489206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:47.512 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:47.538660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:57.606 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:57.635641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:05:57.618 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:05:57.648060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:07.731 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:07.760404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:07.743 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:07.773525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:17.857 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:17.886584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:17.869 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:17.898853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:27.999 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:28.028455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:28.011 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:28.040869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:38.107 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:38.137352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:38.119 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:38.149572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:48.227 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:48.257184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:48.244 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:48.274016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:58.340 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:58.370357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:06:58.355 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:06:58.385279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:08.457 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:08.487031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:08.472 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:08.502573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:18.589 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:18.618723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:18.598 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:18.628890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:28.701 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:28.731053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:28.716 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:28.746823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:38.789 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:38.819028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:38.803 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:38.833005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:48.865 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:48.894486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:48.878 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:48.907801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:58.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:59.016793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:07:59.000 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:07:59.030189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:09.109 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:09.138819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:09.121 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:09.151646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:19.204 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:19.234730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:19.216 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:19.246691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:29.294 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:29.324626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:29.306 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:29.336964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:39.390 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:39.420819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:39.402 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:39.433110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:49.479 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:49.509893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:49.491 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:49.521582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:59.550 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:59.580799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:08:59.565 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:08:59.595466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:09.651 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:09.681395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:09.663 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:09.693563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:19.779 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:19.810165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:19.794 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:19.824399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:29.920 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:29.951140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:29.933 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:29.964174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:40.058 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:40.088739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:40.073 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:40.104216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:50.174 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:50.204462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:09:50.188 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:09:50.219480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:00.333 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:00.364080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:00.346 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:00.377209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:10.452 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:10.482901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:10.464 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:10.495062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:20.539 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:20.570168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:20.552 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:20.583572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:30.648 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:30.678567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:30.663 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:30.693951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:40.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:40.752724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:40.733 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:40.764482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:50.858 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:50.889030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:10:50.871 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:10:50.902582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:01.013 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:01.044731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:01.023 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:01.054611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:11.111 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:11.142091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:11.139 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:11.165294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:21.241 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:21.272399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:21.254 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:21.285279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:31.348 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:31.379253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:31.362 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:31.393809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:41.421 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:41.452768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:41.435 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:41.466905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:51.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:51.527769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:11:51.545 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:11:51.576777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:01.676 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:01.707087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:01.688 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:01.719598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:11.783 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:11.813796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:11.799 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:11.830215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:21.883 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:21.914671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:21.895 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:21.926952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:31.961 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:31.992907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:31.990 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:32.021824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:42.074 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:42.105065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:42.086 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:42.117650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:52.152 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:52.184142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:12:52.166 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:12:52.198050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:02.238 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:02.269585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:02.251 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:02.283189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:12.349 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:12.380562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:12.362 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:12.393567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:22.455 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:22.486692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:22.469 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:22.500708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:32.541 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:32.572892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:32.559 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:32.587274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:42.680 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:42.711557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:42.693 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:42.725228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:52.756 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:52.787973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:13:52.769 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:13:52.800955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:02.916 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:02.948356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:02.928 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:02.960949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:13.047 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:13.079053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:13.058 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:13.090816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:23.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:23.189724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:23.172 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:23.204277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:33.234 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:33.266277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:33.245 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:33.277393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:43.345 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:43.376976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:43.376 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:43.408527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:53.494 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:53.526161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:14:53.509 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:14:53.541805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:03.641 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:03.673461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:03.654 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:03.687147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:13.747 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:13.779876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:13.759 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:13.791751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:23.857 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:23.889372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:23.870 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:23.902692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:34.009 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:34.041496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:34.022 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:34.054738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:44.174 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:44.207027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:44.188 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:44.221574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:54.293 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:54.325841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:15:54.307 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:15:54.339694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:04.376 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:04.409382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:04.391 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:04.423896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:14.512 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:14.544759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:14.530 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:14.563157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:24.612 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:24.645102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:24.661 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:24.694403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:34.741 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:34.773723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:34.796 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:34.828442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:44.863 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:44.895907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:44.883 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:44.916631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:55.091 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:55.123925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:16:55.109 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:16:55.140425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:05.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:05.218324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:05.201 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:05.234596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:15.273 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:15.306578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:15.287 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:15.320414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:25.400 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:25.432986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:25.415 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:25.448534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:35.477 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:35.510080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:35.491 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:35.524344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:45.561 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:45.594622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:45.573 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:45.606803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:51.205 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:51.238225 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T19:17:51.861 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:51.893379 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:17:51.882+0000 7fa1ea173340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:17:51.861 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T19:17:51.882+0000 7fa1ea173340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:17:51.861 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T19:17:51.882+0000 7fa1ea173340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:17:51.861 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:17:55.687 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:55.720637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:17:55.699 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:17:55.732560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:05.771 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:05.804457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:05.818 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:05.851771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:15.879 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:15.912446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:15.891 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:15.925025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:25.961 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:25.994856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:25.974 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:26.008337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:36.046 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:36.079502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:36.057 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:36.091433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:46.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:46.193285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:46.172 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:46.206308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:56.270 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:56.303445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:18:56.281 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:18:56.315395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:06.370 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:06.403839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:06.420 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:06.453975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:16.512 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:16.546143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:16.527 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:16.560990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:26.611 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:26.645437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:26.623 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:26.657672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:36.701 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:36.735028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:36.717 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:36.751454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:46.795 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:46.829813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:46.810 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:46.844114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:56.922 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:56.955766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:19:56.933 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:19:56.968077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:06.996 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:07.030440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:07.009 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:07.044075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:17.127 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:17.161096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:17.141 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:17.175598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:27.247 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:27.281404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:27.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:27.291888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:37.325 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:37.359575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:37.338 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:37.373012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:47.411 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:47.445594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:47.424 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:47.459207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:57.524 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:57.559076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:20:57.586 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:20:57.621137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:07.707 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:07.741385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:07.720 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:07.755141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:17.830 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:17.864382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:17.844 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:17.878674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:27.958 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:27.992358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:27.971 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:28.005939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:38.033 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:38.067780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:38.043 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:38.078285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:48.130 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:48.164803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:48.145 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:48.179669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:58.244 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:58.279287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:21:58.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:21:58.292675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:08.324 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:08.359466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:08.337 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:08.372518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:18.426 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:18.461279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:18.501 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:18.536532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:28.595 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:28.629848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:28.609 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:28.644084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:38.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:38.762186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:38.739 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:38.774862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:48.813 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:48.848025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:48.832 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:48.867103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:58.938 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:58.972923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:22:58.950 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:22:58.985440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:09.053 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:09.088254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:09.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:09.099495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:19.137 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:19.172635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:19.194 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:19.229285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:29.261 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:29.296072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:29.275 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:29.310440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:39.335 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:39.370636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:39.347 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:39.382692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:49.443 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:49.478467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:49.455 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:49.488624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:59.556 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:59.591918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:23:59.566 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:23:59.602244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:09.663 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:09.698311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:09.676 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:09.711975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:19.741 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:19.777176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:19.754 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:19.790169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:29.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:29.861240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:29.837 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:29.873173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:39.905 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:39.941166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:39.917 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:39.953406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:49.988 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:50.017981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:24:49.994 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:24:50.029753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:00.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:00.099247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:00.076 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:00.112934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:10.151 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:10.187102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:10.210 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:10.246621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:20.315 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:20.351456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:20.330 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:20.366378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:30.401 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:30.437161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:30.415 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:30.451166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:40.493 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:40.529316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:40.566 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:40.602781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:50.703 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:50.738782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:25:50.716 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:25:50.752576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:00.811 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:00.846181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:00.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:00.860388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:10.921 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:10.955847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:10.936 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:10.970906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:21.057 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:21.091138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:21.068 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:21.101897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:31.189 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:31.221721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:31.199 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:31.232399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:41.281 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:41.312732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:41.295 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:41.327082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:51.414 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:51.445915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:26:51.427 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:26:51.458885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:01.532 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:01.563009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:01.545 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:01.575545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:11.645 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:11.674944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:11.659 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:11.689082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:21.747 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:21.776501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:21.760 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:21.789921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:31.878 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:31.906511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:31.886 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:31.915707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:42.016 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:42.043834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:42.034 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:42.058351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:52.142 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:52.169908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:27:52.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:27:52.184360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:02.267 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:02.294530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:02.282 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:02.309247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:12.355 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:12.381105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:12.367 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:12.393631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:22.492 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:22.517341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:22.504 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:22.529912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:32.603 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:32.628428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:32.617 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:32.642865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:42.713 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:42.738014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:42.729 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:42.753471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:52.830 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:52.853776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:28:52.845 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:28:52.867804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:02.914 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:02.937495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:02.927 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:02.951009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:13.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:13.087197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:13.080 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:13.102811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:23.154 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:23.175782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:23.167 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:23.189027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:33.286 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:33.307715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:33.307 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:33.328394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:43.412 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:43.433162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:43.427 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:43.448333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:53.591 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:53.612013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:29:53.603 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:29:53.623784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:03.759 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:03.778899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:03.770 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:03.791046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:13.915 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:13.934861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:13.930 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:13.949828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:24.066 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:24.085209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:24.078 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:24.097656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:34.171 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:34.189587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:34.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:34.201688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:44.255 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:44.273071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:44.269 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:44.287264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:54.344 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:54.361745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:30:54.355 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:30:54.372699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:04.423 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:04.439899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:04.436 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:04.453571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:14.529 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:14.546037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:14.540 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:14.556982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:24.628 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:24.644160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:24.639 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:24.655367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:34.716 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:34.732446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:34.765 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:34.780982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:44.874 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:44.889674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:44.892 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:44.902598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:54.994 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:55.009053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:31:55.008 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:31:55.023508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:05.113 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:05.127454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:05.145 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:05.159277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:15.283 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:15.296976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:15.297 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:15.311789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:25.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:25.406524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:25.404 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:25.418213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:35.479 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:35.492049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:35.490 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:35.504015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:45.582 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:45.594975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:45.593 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:45.606220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:55.657 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:55.669886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:32:55.670 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:32:55.683370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:05.741 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:05.752837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:05.756 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:05.767892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:15.826 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:15.838341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:15.838 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:15.850090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:25.935 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:25.946860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:25.950 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:25.961230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:36.059 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:36.069744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:36.072 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:36.083072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:46.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:46.196208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:46.198 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:46.208857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:56.296 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:56.305964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:33:56.309 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:33:56.320237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:06.375 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:06.384936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:06.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:06.403353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:16.504 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:16.513519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:16.516 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:16.526520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:26.629 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:26.638795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:26.641 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:26.651174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:31.897 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:31.906048 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T19:34:32.586 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:32.595064 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:34:32.584+0000 7f20c9b76340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:34:32.587 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T19:34:32.584+0000 7f20c9b76340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:34:32.587 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T19:34:32.584+0000 7f20c9b76340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:34:32.587 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:34:36.707 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:36.716110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:36.720 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:36.729581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:46.819 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:46.827642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:46.833 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:46.842128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:56.900 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:56.907919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:34:56.912 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:34:56.920999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:07.015 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:07.023101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:07.028 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:07.036637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:17.138 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:17.145524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:17.150 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:17.157879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:27.278 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:27.285279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:27.292 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:27.299847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:37.399 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:37.405897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:37.410 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:37.417745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:47.476 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:47.482754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:47.487 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:47.494581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:57.572 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:57.578581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:35:57.583 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:35:57.589686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:07.679 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:07.685853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:07.693 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:07.699497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:17.796 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:17.801954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:17.808 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:17.814198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:27.920 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:27.925739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:27.934 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:27.939669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:38.085 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:38.090672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:38.097 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:38.103002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:48.179 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:48.184612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:48.189 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:48.194288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:58.296 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:58.301469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:36:58.311 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:36:58.316422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:08.453 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:08.457921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:08.467 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:08.471972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:18.543 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:18.547851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:18.557 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:18.561243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:28.638 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:28.642113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:28.652 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:28.656505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:38.750 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:38.753760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:38.763 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:38.767292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:48.838 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:48.841384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:48.850 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:48.854560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:58.952 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:58.954786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:37:58.962 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:37:58.965363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:09.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:09.065944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:09.077 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:09.080787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:19.142 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:19.145330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:19.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:19.159209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:29.243 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:29.246335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:29.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:29.260262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:39.331 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:39.334020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:39.342 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:39.344834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:49.415 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:49.417325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:49.430 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:49.432349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:59.564 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:59.566542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:38:59.576 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:38:59.578807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:09.646 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:09.648241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:09.659 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:09.660657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:19.771 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:19.772198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:19.784 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:19.785770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:29.852 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:29.853605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:29.864 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:29.866413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:39.936 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:39.937587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:39.951 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:39.952232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:50.029 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:50.029547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:39:50.043 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:39:50.044416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:00.145 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:00.145525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:00.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:00.157753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:10.239 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:10.234016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:10.247 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:10.247535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:20.319 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:20.318869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:20.331 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:20.331629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:30.406 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:30.406413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:30.420 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:30.420005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:40.515 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:40.514933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:40.526 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:40.526745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:50.637 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:50.636907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:40:50.648 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:40:50.648516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:00.750 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:00.749875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:00.764 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:00.764121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:10.856 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:10.855464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:10.876 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:10.870356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:20.987 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:20.986136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:21.000 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:20.999607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:31.103 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:31.102053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:31.117 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:31.116521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:41.229 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:41.228065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:41.241 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:41.239724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:51.313 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:51.311927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:41:51.326 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:41:51.324678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:01.404 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:01.402840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:01.436 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:01.434439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:11.546 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:11.545058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:11.558 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:11.556806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:21.623 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:21.621296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:21.635 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:21.633857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:31.739 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:31.736755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:31.752 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:31.750385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:41.868 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:41.865834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:41.881 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:41.879776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:51.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:51.976355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:42:51.992 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:42:51.990310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:02.161 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:02.159148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:02.176 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:02.174011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:12.304 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:12.302039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:12.318 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:12.315198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:22.418 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:22.415283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:22.433 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:22.430606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:32.511 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:32.507748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:32.524 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:32.520728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:42.598 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:42.595404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:42.612 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:42.609698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:52.716 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:52.712622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:43:52.728 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:43:52.725495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:02.835 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:02.832173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:02.848 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:02.844756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:12.928 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:12.924437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:12.943 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:12.939279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:23.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:23.059589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:23.076 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:23.072497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:33.160 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:33.156373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:33.171 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:33.167794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:43.308 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:43.304593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:43.320 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:43.316594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:53.418 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:53.413891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:44:53.450 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:44:53.446134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:03.559 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:03.555254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:03.574 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:03.569758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:13.672 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:13.667568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:13.685 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:13.680728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:23.828 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:23.823685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:23.842 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:23.837307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:33.917 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:33.912860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:33.930 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:33.926048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:44.062 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:44.057407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:44.073 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:44.068401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:54.148 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:54.142609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:45:54.160 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:45:54.156064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:04.248 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:04.243079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:04.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:04.251980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:14.357 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:14.352325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:14.369 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:14.363821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:24.473 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:24.467522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:24.485 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:24.479879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:34.563 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:34.557583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:34.576 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:34.570531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:44.649 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:44.642940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:44.662 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:44.657144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:54.760 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:54.754169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:46:54.774 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:46:54.768021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:04.878 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:04.871816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:04.891 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:04.885651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:14.947 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:14.941486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:14.961 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:14.955075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:25.024 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:25.018612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:25.038 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:25.032118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:35.150 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:35.144100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:35.162 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:35.156528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:45.246 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:45.239732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:45.261 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:45.254510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:55.371 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:55.364376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:47:55.386 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:47:55.379124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:05.461 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:05.454254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:05.475 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:05.468811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:15.610 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:15.603623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:15.624 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:15.617182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:25.731 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:25.724556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:25.743 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:25.736303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:35.874 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:35.867541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:35.908 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:35.901446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:45.964 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:45.956954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:45.978 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:45.971407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:56.117 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:56.109763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:48:56.128 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:48:56.121458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:06.195 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:06.187708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:06.272 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:06.265071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:16.366 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:16.358665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:16.376 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:16.368797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:26.491 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:26.483192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:26.503 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:26.495869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:36.569 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:36.561321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:36.581 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:36.573424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:46.660 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:46.652888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:46.673 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:46.665959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:56.745 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:56.736880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:49:56.758 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:49:56.750769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:06.869 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:06.861081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:06.880 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:06.872232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:16.950 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:16.941853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:16.964 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:16.955712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:27.084 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:27.075567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:27.095 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:27.087687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:37.181 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:37.172433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:37.191 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:37.183241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:47.267 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:47.258342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:47.317 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:47.309226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:57.406 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:57.397706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:50:57.419 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:50:57.411075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:07.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:07.487019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:07.509 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:07.500598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:12.610 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:12.601528 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T19:51:13.290 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:13.280574 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T19:51:13.269+0000 7feb0e877340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T19:51:13.290 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T19:51:13.269+0000 7feb0e877340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T19:51:13.290 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T19:51:13.269+0000 7feb0e877340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T19:51:13.291 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T19:51:17.580 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:17.571771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:17.592 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:17.584029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:27.670 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:27.661292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:27.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:27.713738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:37.835 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:37.825716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:37.864 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:37.840062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:47.921 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:47.912446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:47.983 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:47.974483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:58.060 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:58.050527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:51:58.073 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:51:58.064501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:08.166 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:08.157424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:08.179 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:08.170711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:18.280 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:18.270442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:18.291 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:18.282400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:28.446 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:28.436846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:28.459 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:28.450329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:38.528 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:38.518015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:38.541 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:38.531722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:48.603 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:48.593837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:48.620 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:48.610653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:58.731 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:58.720865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:52:58.743 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:52:58.733592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:08.817 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:08.807674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:08.873 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:08.863567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:18.981 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:18.970852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:18.992 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:18.981861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:29.125 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:29.115446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:29.137 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:29.127684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:39.220 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:39.209810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:39.271 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:39.260510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:49.380 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:49.369535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:49.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:49.383339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:59.487 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:59.476785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:53:59.497 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:53:59.486123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:09.660 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:09.649936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:09.669 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:09.658246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:19.773 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:19.762623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:19.785 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:19.774787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:29.901 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:29.890749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:29.914 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:29.903911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:39.989 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:39.977844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:40.001 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:39.990669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:50.071 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:50.060243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:54:50.083 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:54:50.073161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:00.154 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:00.142839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:00.167 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:00.156576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:10.270 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:10.259129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:10.286 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:10.275073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:20.392 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:20.381226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:20.403 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:20.392287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:30.479 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:30.468025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:30.571 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:30.559858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:40.639 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:40.628083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:40.705 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:40.694390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:50.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:50.813753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:55:50.838 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:55:50.827524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:00.906 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:00.894838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:00.920 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:00.908638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:11.033 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:11.021133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:11.044 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:11.032804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:21.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:21.145368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:21.170 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:21.158358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:31.285 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:31.273671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:31.296 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:31.284953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:41.415 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:41.403117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:41.428 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:41.416406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:51.508 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:51.495851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:56:51.583 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:56:51.571471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:01.717 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:01.704943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:01.731 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:01.719006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:11.829 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:11.816775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:11.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:11.828789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:21.969 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:21.956808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:21.980 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:21.968251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:32.049 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:32.032682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:32.097 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:32.085262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:42.187 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:42.174286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:42.201 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:42.188935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:52.314 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:52.301260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:57:52.331 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:57:52.318763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:02.430 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:02.417356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:02.444 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:02.432170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:12.549 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:12.536405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:12.564 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:12.551886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:22.664 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:22.651781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:22.677 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:22.664424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:32.744 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:32.731439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:32.758 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:32.745243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:42.877 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:42.864388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:42.892 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:42.879799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:52.964 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:52.950545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:58:52.975 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:58:52.962560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:03.095 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:03.082183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:03.108 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:03.095216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:13.175 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:13.161538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:13.291 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:13.277685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:23.385 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:23.372137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:23.399 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:23.385753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:33.467 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:33.453529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:33.479 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:33.465713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:43.590 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:43.577348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:43.605 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:43.591430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:53.708 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:53.694188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T19:59:53.732 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 19:59:53.717606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:03.802 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:03.788993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:03.816 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:03.802777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:13.884 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:13.871160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:13.898 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:13.885113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:23.959 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:23.945870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:23.970 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:23.957056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:34.139 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:34.126303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:34.154 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:34.141890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:44.277 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:44.264275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:44.291 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:44.278838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:54.402 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:54.389897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:00:54.416 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:00:54.403261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:04.513 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:04.500521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:04.527 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:04.514807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:14.638 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:14.626428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:14.653 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:14.640875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:24.780 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:24.767970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:24.788 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:24.776855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:34.873 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:34.861595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:34.887 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:34.875664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:44.965 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:44.952270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:44.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:44.967373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:55.097 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:55.085528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:01:55.107 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:01:55.095792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:05.200 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:05.188535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:05.209 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:05.198154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:15.326 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:15.315451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:15.342 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:15.330976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:25.512 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:25.500914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:25.525 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:25.514649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:35.606 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:35.595704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:35.619 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:35.608928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:45.697 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:45.686151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:45.745 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:45.734652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:55.818 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:55.807836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:02:55.836 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:02:55.825723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:05.920 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:05.910218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:05.985 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:05.974688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:16.079 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:16.069019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:16.094 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:16.084519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:26.197 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:26.186922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:26.209 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:26.199217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:36.335 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:36.325131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:36.361 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:36.351703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:46.515 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:46.505032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:46.528 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:46.518224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:56.589 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:56.579662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:03:56.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:03:56.591606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:06.717 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:06.707284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:06.728 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:06.719004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:16.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:16.787154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:16.809 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:16.799431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:26.871 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:26.862149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:26.884 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:26.874665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:36.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:36.969220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:36.993 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:36.984600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:47.102 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:47.092365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:47.112 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:47.103188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:57.223 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:57.213429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:04:57.238 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:04:57.228738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:07.346 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:07.336633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:07.359 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:07.350728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:17.494 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:17.485181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:17.502 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:17.494037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:27.587 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:27.578441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:27.600 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:27.590931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:37.684 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:37.675450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:37.698 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:37.689850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:47.814 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:47.805058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:47.828 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:47.819776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:57.898 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:57.889991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:05:57.913 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:05:57.904354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:08.024 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:08.015409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:08.037 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:08.028787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:18.105 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:18.096787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:18.118 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:18.110433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:28.188 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:28.180380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:28.199 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:28.190705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:38.278 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:38.269814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:38.292 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:38.284648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:48.377 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:48.368876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:48.391 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:48.383352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:58.492 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:58.484004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:06:58.505 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:06:58.496808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:08.597 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:08.589147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:08.610 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:08.602316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:18.684 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:18.675907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:18.781 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:18.773666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:28.860 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:28.851730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:28.873 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:28.865331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:38.980 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:38.972339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:38.990 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:38.982525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:49.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:49.090100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:49.110 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:49.102158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:53.295 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:53.287433 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T20:07:53.978 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:53.970229 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T20:07:53.958+0000 7fbbec2ba340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T20:07:53.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T20:07:53.958+0000 7fbbec2ba340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T20:07:53.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T20:07:53.958+0000 7fbbec2ba340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T20:07:53.979 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T20:07:59.250 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:59.242253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:07:59.264 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:07:59.256419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:09.339 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:09.331628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:09.353 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:09.345516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:19.454 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:19.446921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:19.468 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:19.460915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:29.612 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:29.605088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:29.631 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:29.623936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:39.712 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:39.704542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:39.722 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:39.714440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:49.797 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:49.789278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:49.809 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:49.802259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:59.954 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:59.946339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:08:59.970 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:08:59.962715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:10.045 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:10.037591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:10.062 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:10.053150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:20.167 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:20.160290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:20.178 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:20.170786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:30.295 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:30.287358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:30.307 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:30.300200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:40.370 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:40.363145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:40.385 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:40.378048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:50.470 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:50.462838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:09:50.482 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:09:50.475175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:00.591 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:00.583539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:00.606 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:00.599292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:10.714 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:10.706947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:10.725 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:10.718055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:20.836 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:20.829111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:20.851 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:20.844405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:30.937 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:30.930263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:30.950 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:30.943628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:41.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:41.054686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:41.073 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:41.066373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:51.205 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:51.197552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:10:51.217 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:10:51.210417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:01.284 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:01.277151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:01.298 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:01.291104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:11.376 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:11.369354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:11.387 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:11.380880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:21.480 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:21.473428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:21.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:21.489291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:31.559 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:31.552495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:31.570 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:31.564039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:41.690 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:41.683045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:41.702 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:41.695850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:51.769 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:51.761965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:11:51.783 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:11:51.776091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:01.884 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:01.877208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:01.896 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:01.889702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:11.983 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:11.976629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:11.997 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:11.990613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:22.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:22.055893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:22.075 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:22.068323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:32.207 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:32.200694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:32.220 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:32.213502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:42.280 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:42.273471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:42.293 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:42.286530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:52.391 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:52.384869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:12:52.405 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:12:52.398585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:02.509 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:02.502766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:02.523 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:02.516785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:12.634 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:12.627135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:12.647 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:12.640502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:22.755 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:22.748365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:22.767 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:22.761197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:32.834 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:32.828062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:32.847 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:32.840483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:42.952 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:42.945138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:42.967 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:42.960179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:53.080 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:53.072999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:13:53.093 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:13:53.086681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:03.157 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:03.150292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:03.218 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:03.211537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:13.307 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:13.300599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:13.322 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:13.315273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:23.435 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:23.428172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:23.451 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:23.444762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:33.518 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:33.511711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:33.531 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:33.525000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:43.615 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:43.608103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:43.627 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:43.621044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:53.698 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:53.690935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:14:53.711 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:14:53.704523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:03.820 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:03.813870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:03.834 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:03.827114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:13.937 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:13.930306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:13.947 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:13.941377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:24.015 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:24.007981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:24.029 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:24.022414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:34.133 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:34.126767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:34.144 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:34.138026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:44.224 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:44.217523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:44.237 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:44.230794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:54.379 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:54.372713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:15:54.391 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:15:54.385213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:04.463 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:04.456274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:04.475 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:04.469095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:14.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:14.594819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:14.614 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:14.607912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:24.712 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:24.705483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:24.774 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:24.767664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:34.833 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:34.826141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:34.844 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:34.838132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:44.945 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:44.938233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:44.961 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:44.954089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:55.025 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:55.018999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:16:55.042 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:16:55.035708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:05.113 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:05.106948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:05.128 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:05.121480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:15.233 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:15.226666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:15.248 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:15.241382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:25.344 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:25.337408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:25.354 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:25.348298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:35.429 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:35.422182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:35.441 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:35.434554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:45.549 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:45.542349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:45.564 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:45.558150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:55.670 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:55.663372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:17:55.682 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:17:55.675545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:05.749 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:05.742107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:05.763 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:05.756469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:15.874 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:15.867497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:15.889 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:15.882610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:25.959 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:25.952246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:25.973 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:25.966269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:36.126 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:36.119122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:36.141 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:36.134541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:46.211 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:46.204721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:46.224 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:46.217522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:56.299 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:56.292334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:18:56.313 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:18:56.306830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:06.417 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:06.409804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:06.429 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:06.422623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:16.516 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:16.509223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:16.529 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:16.522808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:26.601 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:26.593887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:26.613 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:26.606949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:36.693 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:36.685763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:36.705 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:36.698384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:46.805 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:46.797994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:46.819 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:46.812517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:56.886 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:56.879821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:19:56.900 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:19:56.893634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:06.988 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:06.981809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:07.102 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:07.002210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:17.069 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:17.062704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:17.129 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:17.123090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:27.238 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:27.231682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:27.251 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:27.244992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:37.365 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:37.358962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:37.377 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:37.370524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:47.472 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:47.465951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:47.529 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:47.522771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:57.623 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:57.616702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:20:57.634 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:20:57.628089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:07.747 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:07.739935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:07.760 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:07.753782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:17.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:17.818670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:17.837 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:17.830682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:27.912 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:27.905525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:27.923 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:27.916764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:37.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:37.979167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:37.998 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:37.992071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:48.069 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:48.062140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:48.082 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:48.075579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:58.168 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:58.161048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:21:58.183 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:21:58.176459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:08.255 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:08.248385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:08.312 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:08.305777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:18.422 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:18.415703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:18.434 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:18.427953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:28.533 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:28.526258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:28.548 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:28.541537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:38.630 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:38.622856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:38.643 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:38.636694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:48.760 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:48.752681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:48.774 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:48.766772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:58.837 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:58.829768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:22:58.849 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:22:58.841992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:09.017 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:09.009837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:09.030 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:09.023236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:19.151 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:19.143993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:19.166 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:19.159930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:29.271 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:29.263852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:29.283 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:29.276221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:39.355 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:39.347778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:39.368 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:39.361733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:49.499 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:49.492221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:49.512 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:49.505564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:59.591 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:59.584120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:23:59.602 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:23:59.594993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:09.723 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:09.715941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:09.738 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:09.731084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:19.871 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:19.863949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:19.882 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:19.874576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:29.969 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:29.962406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:29.982 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:29.975070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:33.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:33.979238 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T20:24:34.658 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:34.650938 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T20:24:34.642+0000 7f1c8ff24340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T20:24:34.659 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T20:24:34.642+0000 7f1c8ff24340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T20:24:34.659 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T20:24:34.642+0000 7f1c8ff24340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T20:24:34.659 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T20:24:40.101 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:40.093621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:40.111 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:40.104209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:50.220 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:50.212705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:24:50.233 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:24:50.226159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:00.360 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:00.353044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:00.405 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:00.397871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:10.569 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:10.561549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:10.583 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:10.575396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:20.727 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:20.719748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:20.741 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:20.734040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:30.841 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:30.834416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:30.855 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:30.847693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:40.965 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:40.957949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:40.978 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:40.971296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:51.041 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:51.033629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:25:51.054 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:25:51.046664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:01.143 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:01.135740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:01.162 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:01.154622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:11.269 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:11.261664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:11.281 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:11.274219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:21.347 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:21.339829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:21.361 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:21.354072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:31.466 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:31.458277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:31.482 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:31.474058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:41.551 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:41.543773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:41.566 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:41.558495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:51.634 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:51.626544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:26:51.648 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:26:51.640604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:01.730 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:01.722673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:01.745 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:01.737560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:11.809 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:11.801321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:11.824 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:11.816576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:21.941 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:21.932780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:21.956 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:21.949029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:32.030 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:32.022737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:32.049 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:32.041838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:42.118 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:42.110365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:42.132 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:42.124708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:52.213 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:52.205269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:27:52.226 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:27:52.218976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:02.296 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:02.288462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:02.309 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:02.301889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:12.380 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:12.371995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:12.394 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:12.385293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:22.489 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:22.481439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:22.502 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:22.494676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:32.567 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:32.559494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:32.582 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:32.574147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:42.673 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:42.665630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:42.687 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:42.679833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:52.796 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:52.788023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:28:52.810 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:28:52.802611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:02.877 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:02.869477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:02.889 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:02.880796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:12.953 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:12.945492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:12.968 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:12.960532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:23.037 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:23.029446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:23.125 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:23.117213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:33.200 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:33.192296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:33.212 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:33.204207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:43.302 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:43.294352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:43.312 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:43.304629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:53.432 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:53.424236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:29:53.446 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:29:53.438183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:03.558 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:03.549723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:03.570 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:03.562830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:13.692 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:13.684086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:13.704 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:13.696765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:23.808 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:23.800038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:23.822 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:23.813834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:33.933 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:33.925263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:33.944 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:33.936244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:44.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:44.055360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:44.078 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:44.069962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:54.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:54.151421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:30:54.209 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:30:54.201231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:04.339 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:04.330271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:04.349 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:04.340766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:14.417 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:14.409022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:14.432 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:14.424355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:24.525 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:24.516889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:24.537 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:24.529195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:34.647 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:34.638265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:34.659 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:34.650634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:44.767 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:44.758634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:44.781 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:44.772609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:54.930 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:54.921987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:31:54.945 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:31:54.936864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:05.010 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:05.001587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:05.078 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:05.070072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:15.162 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:15.153984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:15.177 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:15.168920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:25.274 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:25.265858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:25.288 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:25.279903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:35.386 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:35.377295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:35.399 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:35.390683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:45.496 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:45.487950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:45.510 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:45.501619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:55.628 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:55.618984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:32:55.665 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:32:55.657033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:05.737 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:05.728519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:05.750 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:05.741424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:15.874 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:15.865493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:15.907 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:15.898287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:25.996 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:25.987543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:26.051 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:26.042747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:36.170 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:36.161504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:36.181 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:36.172930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:46.276 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:46.266979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:46.288 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:46.279606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:56.360 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:56.351154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:33:56.372 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:33:56.363742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:06.444 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:06.434955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:06.457 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:06.448422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:16.561 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:16.552490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:16.574 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:16.565478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:26.649 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:26.640890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:26.665 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:26.657073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:36.790 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:36.782117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:36.799 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:36.791995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:46.903 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:46.896111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:46.916 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:46.909219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:56.985 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:56.978389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:34:56.995 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:34:56.989025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:07.093 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:07.086309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:07.104 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:07.097716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:17.175 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:17.169023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:17.230 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:17.224346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:27.330 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:27.324045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:27.342 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:27.335894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:37.410 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:37.404209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:37.423 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:37.417474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:47.497 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:47.491776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:47.510 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:47.505077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:57.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:57.583082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:35:57.599 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:35:57.594710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:07.728 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:07.721332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:07.737 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:07.733380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:17.850 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:17.846014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:17.863 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:17.858987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:27.966 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:27.962003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:27.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:27.974883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:38.063 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:38.058660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:38.078 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:38.074164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:48.143 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:48.139357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:48.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:48.152865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:58.257 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:58.253227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:36:58.270 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:36:58.267234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:08.387 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:08.384139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:08.399 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:08.396985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:18.511 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:18.508409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:18.525 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:18.522699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:28.633 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:28.631111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:28.648 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:28.645908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:38.755 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:38.753124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:38.771 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:38.769761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:48.864 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:48.862775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:48.878 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:48.876820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:58.990 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:58.989209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:37:59.006 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:37:59.004903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:09.112 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:09.111366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:09.145 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:09.144188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:19.215 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:19.213831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:19.227 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:19.226248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:29.363 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:29.362585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:29.378 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:29.377152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:39.495 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:39.494484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:39.513 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:39.511150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:49.648 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:49.648069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:49.658 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:49.658179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:59.821 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:59.821008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:38:59.864 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:38:59.864498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:09.969 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:09.969420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:09.982 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:09.982004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:20.098 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:20.098199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:20.110 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:20.110910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:30.172 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:30.172839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:30.189 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:30.190462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:40.265 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:40.265619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:40.276 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:40.277397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:50.349 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:50.349891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:39:50.364 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:39:50.365477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:00.433 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:00.433463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:00.446 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:00.447160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:10.543 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:10.544531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:10.557 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:10.558229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:20.658 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:20.659766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:20.671 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:20.673299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:30.747 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:30.748806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:30.759 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:30.761878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:40.858 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:40.860507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:40.870 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:40.872505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:50.963 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:50.965239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:40:50.976 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:40:50.978198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:01.080 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:01.083007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:01.094 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:01.096788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:11.170 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:11.172940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:11.181 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:11.183609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:14.660 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:14.662722 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T20:41:15.342 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:15.344137 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T20:41:15.331+0000 7f9d599d9340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T20:41:15.342 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T20:41:15.331+0000 7f9d599d9340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T20:41:15.342 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T20:41:15.331+0000 7f9d599d9340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T20:41:15.343 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T20:41:21.261 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:21.263888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:21.317 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:21.319891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:31.378 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:31.381438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:31.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:31.396889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:41.480 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:41.482857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:41.493 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:41.496955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:51.570 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:51.573751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:41:51.583 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:41:51.586887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:01.656 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:01.659627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:01.700 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:01.703650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:11.810 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:11.813897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:11.824 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:11.828341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:21.948 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:21.951388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:21.962 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:21.966137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:32.082 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:32.085710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:32.092 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:32.095805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:42.208 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:42.212262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:42.223 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:42.227427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:52.304 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:52.306940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:42:52.353 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:42:52.357248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:02.464 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:02.468934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:02.479 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:02.483599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:12.605 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:12.609386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:12.619 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:12.623508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:22.684 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:22.689027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:22.729 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:22.734178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:32.938 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:32.942679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:32.952 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:32.957356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:43.064 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:43.069037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:43.077 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:43.082284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:53.187 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:53.191718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:43:53.199 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:43:53.204399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:03.273 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:03.277854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:03.321 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:03.326505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:13.419 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:13.424450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:13.446 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:13.438075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:23.534 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:23.539414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:23.546 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:23.551722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:33.659 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:33.664638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:33.675 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:33.680705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:43.799 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:43.804697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:43.817 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:43.823185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:53.927 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:53.933374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:44:53.942 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:44:53.947908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:04.057 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:04.063744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:04.069 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:04.075398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:14.134 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:14.140063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:14.146 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:14.152755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:24.238 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:24.244727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:24.252 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:24.258520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:34.352 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:34.358623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:34.374 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:34.380809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:44.450 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:44.456437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:44.465 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:44.472247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:54.537 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:54.543404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:45:54.548 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:45:54.555201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:04.657 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:04.663877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:04.670 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:04.676587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:14.747 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:14.754143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:14.809 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:14.816653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:24.893 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:24.900408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:24.907 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:24.913949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:34.998 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:35.004779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:35.009 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:35.016196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:45.087 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:45.094440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:45.101 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:45.108043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:55.180 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:55.187549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:46:55.266 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:46:55.273256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:05.366 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:05.373757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:05.382 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:05.389662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:15.452 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:15.459183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:15.511 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:15.518435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:25.619 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:25.626402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:25.631 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:25.638377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:35.701 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:35.708409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:35.713 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:35.721418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:45.804 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:45.811676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:45.814 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:45.822668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:55.897 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:55.904508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:47:55.909 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:47:55.917319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:05.979 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:05.987006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:05.990 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:05.998113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:16.153 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:16.161206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:16.165 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:16.173077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:26.270 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:26.277631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:26.281 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:26.289179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:36.358 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:36.366456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:36.372 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:36.380399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:46.450 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:46.458662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:46.463 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:46.471965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:56.566 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:56.574112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:48:56.578 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:48:56.587093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:06.691 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:06.699797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:06.703 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:06.711761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:16.777 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:16.785649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:16.788 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:16.796811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:26.867 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:26.875212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:26.879 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:26.888117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:36.949 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:36.957388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:36.960 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:36.968474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:47.068 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:47.077239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:47.083 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:47.091518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:57.156 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:57.164255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:49:57.217 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:49:57.226054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:07.345 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:07.354085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:07.358 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:07.367017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:17.639 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:17.648108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:17.652 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:17.661542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:27.755 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:27.763756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:27.767 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:27.776725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:37.838 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:37.847202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:37.887 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:37.895694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:47.988 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:47.997354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:48.002 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:48.011837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:58.074 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:58.083113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:50:58.086 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:50:58.095861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:08.185 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:08.194660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:08.199 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:08.208966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:18.298 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:18.307604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:18.311 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:18.320045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:28.375 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:28.384029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:28.392 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:28.401886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:38.457 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:38.466737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:38.513 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:38.522817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:48.614 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:48.623341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:48.627 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:48.636686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:58.721 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:58.730026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:51:58.734 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:51:58.743937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:08.844 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:08.853701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:08.856 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:08.866236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:18.969 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:18.978421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:18.982 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:18.991427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:29.104 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:29.113513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:29.117 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:29.126679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:39.233 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:39.243011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:39.297 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:39.306846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:49.405 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:49.415176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:49.416 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:49.425858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:59.497 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:59.507118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:52:59.510 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:52:59.519585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:09.579 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:09.589121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:09.593 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:09.602843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:19.653 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:19.663685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:19.754 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:19.764672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:29.822 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:29.831745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:29.834 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:29.843814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:39.919 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:39.929327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:39.930 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:39.939828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:50.077 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:50.087424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:53:50.093 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:53:50.103616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:00.159 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:00.169330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:00.171 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:00.181435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:10.229 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:10.238898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:10.239 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:10.249192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:20.314 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:20.324249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:20.330 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:20.340846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:30.402 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:30.412768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:30.459 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:30.470164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:40.568 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:40.578244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:40.580 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:40.591285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:50.662 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:50.672308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:54:50.716 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:54:50.726813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:00.825 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:00.835794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:00.838 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:00.849149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:10.946 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:10.956757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:10.960 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:10.971441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:21.068 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:21.078425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:21.083 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:21.093671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:31.186 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:31.196323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:31.199 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:31.209676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:41.260 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:41.270673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:41.325 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:41.335277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:51.392 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:51.403112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:55:51.404 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:55:51.415137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:01.486 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:01.496644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:01.497 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:01.508142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:11.582 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:11.592747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:11.594 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:11.605018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:21.689 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:21.699727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:21.704 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:21.714453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:31.819 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:31.830140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:31.833 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:31.843932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:41.903 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:41.914153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:41.917 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:41.928152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:52.011 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:52.022370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:56:52.026 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:56:52.037212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:02.128 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:02.138571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:02.139 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:02.150098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:12.230 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:12.241243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:12.244 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:12.255479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:22.353 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:22.364268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:22.368 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:22.379430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:32.486 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:32.497257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:32.499 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:32.510274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:42.565 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:42.576294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:42.577 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:42.588679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:52.683 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:52.693893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:52.696 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:52.707342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:57:55.354 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:55.365144 I | op-mon: parsing mon endpoints: c=10.109.91.196:6789,a=10.110.173.53:6789,b=10.106.73.178:6789 2022-02-21T20:57:56.038 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:57:56.049261 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-02-21T20:57:56.036+0000 7fe6dcb80340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-02-21T20:57:56.039 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T20:57:56.036+0000 7fe6dcb80340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-02-21T20:57:56.039 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21T20:57:56.036+0000 7fe6dcb80340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-02-21T20:57:56.039 INFO:tasks.rook.operator.smithi005.stdout:couldn't init storage provider. : exit status 5 2022-02-21T20:58:02.814 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:02.825147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:02.828 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:02.839763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:12.968 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:12.979816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:12.981 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:12.992129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:23.088 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:23.099542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:23.102 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:23.112667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:33.226 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:33.236953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:33.238 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:33.249095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:43.309 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:43.320610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:43.322 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:43.333272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:53.417 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:53.428818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:58:53.427 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:58:53.438912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:03.499 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:03.510227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:03.508 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:03.519951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:13.573 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:13.584684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:13.588 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:13.598906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:23.704 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:23.715465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:23.715 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:23.726608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:33.816 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:33.827848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:33.830 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:33.841550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:43.940 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:43.951584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:43.952 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:43.964270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:54.078 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:54.090371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T20:59:54.090 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 20:59:54.102228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:04.162 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:04.173722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:04.177 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:04.188123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:14.297 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:14.308125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:14.313 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:14.324520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:24.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:24.402810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:24.456 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:24.468033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:34.539 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:34.551121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:34.557 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:34.565203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:44.631 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:44.641558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:44.642 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:44.654294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:54.755 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:54.766795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:00:54.768 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:00:54.780218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:04.855 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:04.854814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:04.856 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:04.867646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:14.987 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:14.998856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:15.002 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:15.012417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:25.079 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:25.090164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:25.091 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:25.102030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:35.197 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:35.207075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:35.215 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:35.218801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:45.291 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:45.301493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:45.304 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:45.315814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:55.373 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:55.384445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:01:55.391 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:01:55.398627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:05.485 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:05.491037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:05.501 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:05.505541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:15.571 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:15.582624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:15.654 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:15.663837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:25.775 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:25.785856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:25.832 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:25.814857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:35.929 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:35.917922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:35.929 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:35.932081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:46.062 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:46.074033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:46.079 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:46.088544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:56.238 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:56.242929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:02:56.252 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:02:56.256769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:06.374 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:06.384872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:06.393 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:06.400419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:16.475 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:16.484359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:16.508 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:16.495386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:26.635 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:26.639180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:26.653 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:26.651743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:36.761 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:36.731532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:36.761 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:36.747094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:46.894 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:46.848534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:46.899 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:46.861034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:56.950 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:56.962019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:03:56.967 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:03:56.972639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:07.125 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:07.131033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:07.138 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:07.143550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:17.280 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:17.276780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:17.280 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:17.289754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:27.398 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:27.401479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:27.408 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:27.413986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:37.486 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:37.496694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:37.501 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:37.510455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:47.590 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:47.599042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:47.611 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:47.608412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:57.818 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:57.718119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:04:57.819 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:04:57.730485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:07.795 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:07.803006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:07.811 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:07.815892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:17.928 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:17.916378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:17.929 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:17.928441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:28.039 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:28.042229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:28.052 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:28.052647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:38.108 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:38.121005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:38.122 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:38.134640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:48.253 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:48.264903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:48.266 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:48.278551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:58.403 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:58.414826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:05:58.418 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:05:58.429009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:08.533 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:08.541132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:08.543 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:08.555607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:18.619 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:18.626032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:18.631 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:18.638819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:28.746 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:28.758090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:28.759 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:28.770135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:38.866 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:38.874974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:38.877 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:38.886500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:48.986 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:48.998430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:48.999 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:49.012244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:59.079 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:59.091763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:06:59.141 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:06:59.153296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:09.223 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:07:09.235752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T21:07:09.237 INFO:tasks.rook.operator.smithi005.stdout:2022-02-21 21:07:09.250060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-02-21T23:28:33.329 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-02-21T23:28:33.373 DEBUG:teuthology.task.console_log:Killing console logger for smithi005 2022-02-21T23:28:33.374 DEBUG:teuthology.task.console_log:Killing console logger for smithi033 2022-02-21T23:28:33.375 DEBUG:teuthology.task.console_log:Killing console logger for smithi067 2022-02-21T23:28:33.375 DEBUG:teuthology.exit:Finished running handlers