2022-03-21T14:41:13.068 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-21T14:41:13.070 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2022-03-21_14:18:49-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2022-03-21_14:18:49-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6750328 --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} --verbose -- /home/teuthworker/archive/yuriw-2022-03-21_14:18:49-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6750328/orig.config.yaml 2022-03-21T14:41:13.102 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-21T14:41:13.165 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-21_14:18:49-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6750328 branch: wip-yuri6-testing-2022-03-18-1104 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} email: null first_in_suite: false job_id: '6750328' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-21_14:18:49-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri6-testing-2022-03-18-1104 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri6-testing-2022-03-18-1104 sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri6-testing-2022-03-18-1104 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e targets: smithi043.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDI9r+gQt4qvP/YGzkflOK5vVB78NYqrx3N4sVI+V909QKDVk1u7D1C/qcNse+4DkgdGbiBsKaR/ioC5fN09ori+vIvRitz94SJN9L50rFaMeR/R+qVa0362zwHclB4RmSLgHLjWLpfjCfzQNYzWjDdehfUga2FT1S7UM/OUo9zRxU/c55xned8zugLCcCx92AqMX4+H5pHrlcuMa/LWVXUmbXjDo18SN88Umd8Oyc2ZzajjHQ3J84EXXyqGUFBePPez9TLaFReF4sJoDv1A6FGnfjZK86NKHsjgW7O5XO2YsgnL/4g/gV/0AjBIE6HqwRklhdU56ANQfVkCxVMT93baZuYH8brP4gSCQwWBZccxX7jYv+pHx49Z4YEURm/OIELPIcEf0ySEYaHjNa2e6ItVjHiOYCqfjnIek+a4pXLAt2Dddvpg8+ZGFYJdBQQLrQbJtRkLFld3IvQVyAabcrogtN2od+pqxpsyTMCi5TkKo8CeNc7tllDH87rLKPzda0= smithi065.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOoEEh807puo1Myk/IufuQSNKZj8Wqs4FxG1t88dePjRj6iA69VUW+t7hMEM0cGIx+TeSGdVbedo5/Qx0i8n3KcsCNo/DJQq7sgw6xAErdfvxthRWmO4gQsi9174G58i0UszwfPjDNsbKM2FbJ3BY+eS9bxVNgjPKpGxw2BLOlqqlvnMHmyXyQ7Z97naRtPjR8GB+oBE3kei6vooxq0lwwab67vF/ToeOE76wHV6WE7SKFilz2mgzh0oIwIOrvEGm7v76oFey2agPxqW9haoQWCgYnA2JKoquamAV19krXbzSjw4vXUNxy6/bBTWK+Gn3m3Y97mUCobTJWzJTxWCwbMNSuw+JDoizImL1IBCyKhs653sdX65xBAZ2nht40pPEyBLMbEXTpRYS9ecq4iatI2XjO1VJhClBvQ22mwfeD0IJDNEtOKyRt/06KSztOP88/264m2EpYVg1OZaZqH+7kH4hXgM+xCxFx+Gh/vUAAd78E8RNYVucXkTaUV7GHcx8= smithi154.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7oZUdtIjiT27VJ+gZ1HLlWtcsrN28xq/dXv18wRreue4F4g5NKsbrtxWgyHdgDEiEUTRxdCF3xiYo3ShUufK/xDEGuQl8gUE/9HEsZxH/0+MnF62w1XluQE5dcb90av+1ffs/atfgvKnUx1FS/owjQikN31ZYqGJG8lr32bMQy+utoILurif2Yaw1S4Gyb+ZsXbDhJQzReHN0zEROMHost+C9KncWTQ3vAPoma5PGZIQH0fjFKf6avUr0xipeMeBvecmatfq4tcJNHEVzYVM/Kkr6vjy08JmJ9eVwSCgHIxZKRzwj6tiMN0EYs4Z/kYkYIVD3OMHJ98obHN1ggEUsR84gNsnwmMRqHYMivrOxoNhGrSOYeU7WpEJ7iVObUmf0HxChdj/UVMUVjGQdeNcVaiE3Cm7mLGdq4WO1l6Kgzmr5l9A096npmZ6akrZNehUdJfHd1c9s5mgbFj1XRmHc42oIgIsKXz9YQAwgeKmVkDHNO/v69vybvvl2cShCOek= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: a4e18662e5de936eb0ac089e531b933ecbcadb0a timestamp: 2022-03-21_14:18:49 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-21T14:41:13.165 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa; will attempt to use it 2022-03-21T14:41:13.166 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks 2022-03-21T14:41:13.167 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-21T14:41:13.168 INFO:teuthology.task.internal:Checking packages... 2022-03-21T14:41:13.187 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e' 2022-03-21T14:41:13.188 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-21T14:41:13.188 INFO:teuthology.packaging:ref: None 2022-03-21T14:41:13.188 INFO:teuthology.packaging:tag: None 2022-03-21T14:41:13.189 INFO:teuthology.packaging:branch: wip-yuri6-testing-2022-03-18-1104 2022-03-21T14:41:13.189 INFO:teuthology.packaging:sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-21T14:41:13.189 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri6-testing-2022-03-18-1104 2022-03-21T14:41:13.350 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11187-ge92561bf-1focal 2022-03-21T14:41:13.350 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-21T14:41:13.361 INFO:teuthology.task.internal:no buildpackages task found 2022-03-21T14:41:13.362 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-21T14:41:13.449 INFO:teuthology.task.internal:Saving configuration 2022-03-21T14:41:13.461 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-21T14:41:13.522 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-21T14:41:13.546 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-21_14:18:49-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6750328', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-21 14:35:39.127020', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIECn4nVCcGJTth0GTrizPP7OCkV7qkmHnbBo7O7ZvgPe'} 2022-03-21T14:41:13.567 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi065.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-21_14:18:49-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6750328', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-21 14:35:39.127977', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICcK4ysRK0YCIX1XkI9jmeKVBy+M1jBQVTXZthvJujdJ'} 2022-03-21T14:41:13.588 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-21_14:18:49-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6750328', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-21 14:35:39.125560', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMlrA80olX3ENRYQIm08MPaJ2uidx4VA5xYci7JHMZtP'} 2022-03-21T14:41:13.589 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-21T14:41:13.599 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-21T14:41:13.600 INFO:teuthology.task.internal:roles: ubuntu@smithi065.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-21T14:41:13.600 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-21T14:41:13.600 INFO:teuthology.run_tasks:Running task console_log... 2022-03-21T14:41:13.728 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3254514c80>, signals=[15]) 2022-03-21T14:41:13.729 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-21T14:41:13.740 INFO:teuthology.task.internal:Opening connections... 2022-03-21T14:41:13.740 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2022-03-21T14:41:13.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:41:13.796 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi065.front.sepia.ceph.com 2022-03-21T14:41:13.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi065.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:41:13.848 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2022-03-21T14:41:13.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:41:13.905 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-21T14:41:13.916 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2022-03-21T14:41:13.923 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2022-03-21T14:41:13.924 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2022-03-21T14:41:13.972 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="Ubuntu" 2022-03-21T14:41:13.972 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-21T14:41:13.972 INFO:teuthology.orchestra.run.smithi043.stdout:ID=ubuntu 2022-03-21T14:41:13.972 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE=debian 2022-03-21T14:41:13.973 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-21T14:41:13.973 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="20.04" 2022-03-21T14:41:13.973 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-21T14:41:13.973 INFO:teuthology.orchestra.run.smithi043.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-21T14:41:13.974 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-21T14:41:13.974 INFO:teuthology.orchestra.run.smithi043.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-21T14:41:13.974 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_CODENAME=focal 2022-03-21T14:41:13.974 INFO:teuthology.orchestra.run.smithi043.stdout:UBUNTU_CODENAME=focal 2022-03-21T14:41:13.975 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2022-03-21T14:41:13.993 DEBUG:teuthology.orchestra.run.smithi065:> uname -m 2022-03-21T14:41:14.000 INFO:teuthology.orchestra.run.smithi065.stdout:x86_64 2022-03-21T14:41:14.000 DEBUG:teuthology.orchestra.run.smithi065:> cat /etc/os-release 2022-03-21T14:41:14.048 INFO:teuthology.orchestra.run.smithi065.stdout:NAME="Ubuntu" 2022-03-21T14:41:14.048 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-21T14:41:14.048 INFO:teuthology.orchestra.run.smithi065.stdout:ID=ubuntu 2022-03-21T14:41:14.048 INFO:teuthology.orchestra.run.smithi065.stdout:ID_LIKE=debian 2022-03-21T14:41:14.049 INFO:teuthology.orchestra.run.smithi065.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-21T14:41:14.049 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION_ID="20.04" 2022-03-21T14:41:14.049 INFO:teuthology.orchestra.run.smithi065.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-21T14:41:14.049 INFO:teuthology.orchestra.run.smithi065.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-21T14:41:14.049 INFO:teuthology.orchestra.run.smithi065.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-21T14:41:14.050 INFO:teuthology.orchestra.run.smithi065.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-21T14:41:14.050 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION_CODENAME=focal 2022-03-21T14:41:14.050 INFO:teuthology.orchestra.run.smithi065.stdout:UBUNTU_CODENAME=focal 2022-03-21T14:41:14.051 INFO:teuthology.lock.ops:Updating smithi065.front.sepia.ceph.com on lock server 2022-03-21T14:41:14.067 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2022-03-21T14:41:14.075 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2022-03-21T14:41:14.075 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2022-03-21T14:41:14.123 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Ubuntu" 2022-03-21T14:41:14.123 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-21T14:41:14.123 INFO:teuthology.orchestra.run.smithi154.stdout:ID=ubuntu 2022-03-21T14:41:14.124 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE=debian 2022-03-21T14:41:14.124 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-21T14:41:14.124 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="20.04" 2022-03-21T14:41:14.124 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-21T14:41:14.124 INFO:teuthology.orchestra.run.smithi154.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-21T14:41:14.125 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-21T14:41:14.125 INFO:teuthology.orchestra.run.smithi154.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-21T14:41:14.125 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_CODENAME=focal 2022-03-21T14:41:14.125 INFO:teuthology.orchestra.run.smithi154.stdout:UBUNTU_CODENAME=focal 2022-03-21T14:41:14.126 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2022-03-21T14:41:14.146 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-21T14:41:14.158 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-21T14:41:14.192 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-21T14:41:14.193 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2022-03-21T14:41:14.195 DEBUG:teuthology.orchestra.run.smithi065:> test '!' -e /home/ubuntu/cephtest 2022-03-21T14:41:14.198 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2022-03-21T14:41:14.204 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-21T14:41:14.306 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-21T14:41:14.306 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2022-03-21T14:41:14.309 DEBUG:teuthology.orchestra.run.smithi065:> test -z $(ls -A /var/lib/ceph) 2022-03-21T14:41:14.311 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2022-03-21T14:41:14.326 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-21T14:41:14.406 INFO:teuthology.run_tasks:Running task kernel... 2022-03-21T14:41:14.424 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-21T14:41:14.425 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-21T14:41:14.425 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-21T14:41:14.425 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-21T14:41:14.426 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-21T14:41:14.427 DEBUG:teuthology.orchestra.run.smithi065:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-21T14:41:14.427 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-21T14:41:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T14:41:14.433 DEBUG:teuthology.orchestra.run.smithi065:> uname -r 2022-03-21T14:41:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T14:41:14.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-21T14:41:14.436 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2022-03-21T14:41:14.436 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2022-03-21T14:41:14.443 INFO:teuthology.orchestra.run.smithi043.stdout:5.4.0-97-generic 2022-03-21T14:41:14.443 INFO:teuthology.task.kernel:Running kernel on smithi043: 5.4.0-97-generic 2022-03-21T14:41:14.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-get clean 2022-03-21T14:41:14.480 INFO:teuthology.orchestra.run.smithi065.stdout:5.4.0-97-generic 2022-03-21T14:41:14.480 INFO:teuthology.task.kernel:Running kernel on smithi065: 5.4.0-97-generic 2022-03-21T14:41:14.480 DEBUG:teuthology.orchestra.run.smithi065:> sudo apt-get clean 2022-03-21T14:41:14.483 INFO:teuthology.orchestra.run.smithi154.stdout:5.4.0-97-generic 2022-03-21T14:41:14.483 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.4.0-97-generic 2022-03-21T14:41:14.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get clean 2022-03-21T14:41:14.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get update 2022-03-21T14:41:14.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-get update 2022-03-21T14:41:14.568 DEBUG:teuthology.orchestra.run.smithi065:> sudo apt-get update 2022-03-21T14:41:14.716 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:41:14.717 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-21T14:41:14.717 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-21T14:41:14.720 INFO:teuthology.orchestra.run.smithi065.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-21T14:41:14.787 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-21T14:41:14.824 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:41:14.852 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:41:14.852 INFO:teuthology.orchestra.run.smithi043.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-21T14:41:14.904 INFO:teuthology.orchestra.run.smithi065.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-21T14:41:14.913 INFO:teuthology.orchestra.run.smithi043.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-21T14:41:15.024 INFO:teuthology.orchestra.run.smithi065.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-21T14:41:15.101 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-21T14:41:15.116 INFO:teuthology.orchestra.run.smithi065.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-21T14:41:15.157 INFO:teuthology.orchestra.run.smithi065.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-21T14:41:15.160 INFO:teuthology.orchestra.run.smithi065.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-21T14:41:15.160 INFO:teuthology.orchestra.run.smithi065.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-21T14:41:15.173 INFO:teuthology.orchestra.run.smithi065.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-21T14:41:15.174 INFO:teuthology.orchestra.run.smithi065.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-21T14:41:15.176 INFO:teuthology.orchestra.run.smithi065.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-21T14:41:15.177 INFO:teuthology.orchestra.run.smithi065.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-21T14:41:15.184 INFO:teuthology.orchestra.run.smithi065.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-21T14:41:15.192 INFO:teuthology.orchestra.run.smithi065.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-21T14:41:15.193 INFO:teuthology.orchestra.run.smithi065.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-21T14:41:15.205 INFO:teuthology.orchestra.run.smithi065.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-21T14:41:15.206 INFO:teuthology.orchestra.run.smithi065.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-21T14:41:15.207 INFO:teuthology.orchestra.run.smithi065.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-21T14:41:15.207 INFO:teuthology.orchestra.run.smithi065.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-21T14:41:15.219 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-21T14:41:15.226 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-21T14:41:15.230 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-21T14:41:15.230 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-21T14:41:15.238 INFO:teuthology.orchestra.run.smithi065.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-21T14:41:15.239 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-21T14:41:15.239 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-21T14:41:15.239 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-21T14:41:15.240 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-21T14:41:15.241 INFO:teuthology.orchestra.run.smithi154.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-21T14:41:15.245 INFO:teuthology.orchestra.run.smithi043.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-21T14:41:15.249 INFO:teuthology.orchestra.run.smithi154.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-21T14:41:15.258 INFO:teuthology.orchestra.run.smithi154.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-21T14:41:15.259 INFO:teuthology.orchestra.run.smithi154.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-21T14:41:15.271 INFO:teuthology.orchestra.run.smithi154.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-21T14:41:15.271 INFO:teuthology.orchestra.run.smithi154.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-21T14:41:15.272 INFO:teuthology.orchestra.run.smithi154.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-21T14:41:15.272 INFO:teuthology.orchestra.run.smithi154.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-21T14:41:15.319 INFO:teuthology.orchestra.run.smithi043.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-21T14:41:15.333 INFO:teuthology.orchestra.run.smithi154.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-21T14:41:15.342 INFO:teuthology.orchestra.run.smithi154.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-21T14:41:15.346 INFO:teuthology.orchestra.run.smithi154.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-21T14:41:15.347 INFO:teuthology.orchestra.run.smithi154.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-21T14:41:15.353 INFO:teuthology.orchestra.run.smithi154.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-21T14:41:15.353 INFO:teuthology.orchestra.run.smithi154.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-21T14:41:15.355 INFO:teuthology.orchestra.run.smithi154.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-21T14:41:15.355 INFO:teuthology.orchestra.run.smithi154.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-21T14:41:15.365 INFO:teuthology.orchestra.run.smithi154.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-21T14:41:15.366 INFO:teuthology.orchestra.run.smithi065.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-21T14:41:15.372 INFO:teuthology.orchestra.run.smithi154.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-21T14:41:15.374 INFO:teuthology.orchestra.run.smithi154.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-21T14:41:15.377 INFO:teuthology.orchestra.run.smithi154.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-21T14:41:15.378 INFO:teuthology.orchestra.run.smithi154.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-21T14:41:15.378 INFO:teuthology.orchestra.run.smithi154.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-21T14:41:15.378 INFO:teuthology.orchestra.run.smithi154.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-21T14:41:15.379 INFO:teuthology.orchestra.run.smithi154.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-21T14:41:15.389 INFO:teuthology.orchestra.run.smithi154.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-21T14:41:15.389 INFO:teuthology.orchestra.run.smithi154.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-21T14:41:15.408 INFO:teuthology.orchestra.run.smithi154.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-21T14:41:15.409 INFO:teuthology.orchestra.run.smithi154.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-21T14:41:15.410 INFO:teuthology.orchestra.run.smithi154.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-21T14:41:15.410 INFO:teuthology.orchestra.run.smithi154.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-21T14:41:15.469 INFO:teuthology.orchestra.run.smithi043.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-21T14:41:15.612 INFO:teuthology.orchestra.run.smithi065.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-21T14:41:15.712 INFO:teuthology.orchestra.run.smithi043.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-21T14:41:15.714 INFO:teuthology.orchestra.run.smithi043.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-21T14:41:15.727 INFO:teuthology.orchestra.run.smithi043.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-21T14:41:15.736 INFO:teuthology.orchestra.run.smithi043.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-21T14:41:15.737 INFO:teuthology.orchestra.run.smithi043.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-21T14:41:15.738 INFO:teuthology.orchestra.run.smithi043.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-21T14:41:15.750 INFO:teuthology.orchestra.run.smithi065.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-21T14:41:15.760 INFO:teuthology.orchestra.run.smithi065.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-21T14:41:15.761 INFO:teuthology.orchestra.run.smithi065.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-21T14:41:15.793 INFO:teuthology.orchestra.run.smithi043.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-21T14:41:15.797 INFO:teuthology.orchestra.run.smithi065.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-21T14:41:15.798 INFO:teuthology.orchestra.run.smithi065.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-21T14:41:15.802 INFO:teuthology.orchestra.run.smithi043.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-21T14:41:15.803 INFO:teuthology.orchestra.run.smithi043.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-21T14:41:15.804 INFO:teuthology.orchestra.run.smithi065.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-21T14:41:15.804 INFO:teuthology.orchestra.run.smithi065.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-21T14:41:15.830 INFO:teuthology.orchestra.run.smithi043.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-21T14:41:15.831 INFO:teuthology.orchestra.run.smithi065.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-21T14:41:15.848 INFO:teuthology.orchestra.run.smithi043.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-21T14:41:15.849 INFO:teuthology.orchestra.run.smithi065.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-21T14:41:15.852 INFO:teuthology.orchestra.run.smithi043.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-21T14:41:15.852 INFO:teuthology.orchestra.run.smithi043.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-21T14:41:15.853 INFO:teuthology.orchestra.run.smithi065.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-21T14:41:15.865 INFO:teuthology.orchestra.run.smithi043.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-21T14:41:15.866 INFO:teuthology.orchestra.run.smithi043.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-21T14:41:15.873 INFO:teuthology.orchestra.run.smithi043.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-21T14:41:15.883 INFO:teuthology.orchestra.run.smithi065.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-21T14:41:15.884 INFO:teuthology.orchestra.run.smithi065.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-21T14:41:15.885 INFO:teuthology.orchestra.run.smithi065.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-21T14:41:15.886 INFO:teuthology.orchestra.run.smithi065.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-21T14:41:15.886 INFO:teuthology.orchestra.run.smithi065.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-21T14:41:15.891 INFO:teuthology.orchestra.run.smithi043.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-21T14:41:15.891 INFO:teuthology.orchestra.run.smithi043.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-21T14:41:15.892 INFO:teuthology.orchestra.run.smithi043.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-21T14:41:15.892 INFO:teuthology.orchestra.run.smithi043.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-21T14:41:15.906 INFO:teuthology.orchestra.run.smithi043.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-21T14:41:15.911 INFO:teuthology.orchestra.run.smithi043.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-21T14:41:15.911 INFO:teuthology.orchestra.run.smithi043.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-21T14:41:15.911 INFO:teuthology.orchestra.run.smithi065.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-21T14:41:15.935 INFO:teuthology.orchestra.run.smithi043.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-21T14:41:15.958 INFO:teuthology.orchestra.run.smithi043.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-21T14:41:15.960 INFO:teuthology.orchestra.run.smithi043.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-21T14:41:15.991 INFO:teuthology.orchestra.run.smithi043.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-21T14:41:15.992 INFO:teuthology.orchestra.run.smithi043.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-21T14:41:15.992 INFO:teuthology.orchestra.run.smithi043.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-21T14:41:15.993 INFO:teuthology.orchestra.run.smithi065.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-21T14:41:15.993 INFO:teuthology.orchestra.run.smithi065.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-21T14:41:15.993 INFO:teuthology.orchestra.run.smithi065.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-21T14:41:15.994 INFO:teuthology.orchestra.run.smithi065.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-21T14:41:15.995 INFO:teuthology.orchestra.run.smithi065.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-21T14:41:15.995 INFO:teuthology.orchestra.run.smithi043.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-21T14:41:15.995 INFO:teuthology.orchestra.run.smithi043.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-21T14:41:16.015 INFO:teuthology.orchestra.run.smithi043.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-21T14:41:16.095 INFO:teuthology.orchestra.run.smithi043.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-21T14:41:16.095 INFO:teuthology.orchestra.run.smithi043.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-21T14:41:16.096 INFO:teuthology.orchestra.run.smithi043.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-21T14:41:16.097 INFO:teuthology.orchestra.run.smithi043.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-21T14:41:16.097 INFO:teuthology.orchestra.run.smithi043.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-21T14:41:18.257 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 10.2 MB in 2s (6,028 kB/s) 2022-03-21T14:41:18.419 INFO:teuthology.orchestra.run.smithi065.stdout:Fetched 10.2 MB in 2s (6,047 kB/s) 2022-03-21T14:41:18.906 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 10.2 MB in 2s (4,763 kB/s) 2022-03-21T14:41:19.458 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2022-03-21T14:41:19.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-21T14:41:19.551 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2022-03-21T14:41:19.618 INFO:teuthology.orchestra.run.smithi065.stdout:Reading package lists... 2022-03-21T14:41:19.643 DEBUG:teuthology.orchestra.run.smithi065:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-21T14:41:19.708 INFO:teuthology.orchestra.run.smithi065.stdout:Reading package lists... 2022-03-21T14:41:19.723 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2022-03-21T14:41:19.724 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2022-03-21T14:41:19.868 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2022-03-21T14:41:19.868 INFO:teuthology.orchestra.run.smithi154.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-21T14:41:19.869 INFO:teuthology.orchestra.run.smithi154.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-21T14:41:19.869 INFO:teuthology.orchestra.run.smithi154.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-21T14:41:19.869 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2022-03-21T14:41:19.870 INFO:teuthology.orchestra.run.smithi154.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-21T14:41:19.870 INFO:teuthology.orchestra.run.smithi154.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-21T14:41:19.887 INFO:teuthology.orchestra.run.smithi065.stdout:Building dependency tree... 2022-03-21T14:41:19.887 INFO:teuthology.orchestra.run.smithi065.stdout:Reading state information... 2022-03-21T14:41:19.930 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2022-03-21T14:41:19.931 INFO:teuthology.orchestra.run.smithi154.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-21T14:41:19.931 INFO:teuthology.orchestra.run.smithi154.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-21T14:41:19.931 INFO:teuthology.orchestra.run.smithi154.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-21T14:41:19.931 INFO:teuthology.orchestra.run.smithi154.stdout: thermald upower usbmuxd 2022-03-21T14:41:19.988 INFO:teuthology.orchestra.run.smithi154.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-21T14:41:19.989 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 80.1 MB of archives. 2022-03-21T14:41:19.989 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-21T14:41:19.989 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-21T14:41:20.046 INFO:teuthology.orchestra.run.smithi065.stdout:The following additional packages will be installed: 2022-03-21T14:41:20.046 INFO:teuthology.orchestra.run.smithi065.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-21T14:41:20.046 INFO:teuthology.orchestra.run.smithi065.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-21T14:41:20.046 INFO:teuthology.orchestra.run.smithi065.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-21T14:41:20.047 INFO:teuthology.orchestra.run.smithi065.stdout:Suggested packages: 2022-03-21T14:41:20.047 INFO:teuthology.orchestra.run.smithi065.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-21T14:41:20.047 INFO:teuthology.orchestra.run.smithi065.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-21T14:41:20.057 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-21T14:41:20.064 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-21T14:41:20.069 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-21T14:41:20.087 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-21T14:41:20.095 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2022-03-21T14:41:20.096 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-21T14:41:20.102 INFO:teuthology.orchestra.run.smithi065.stdout:The following NEW packages will be installed: 2022-03-21T14:41:20.103 INFO:teuthology.orchestra.run.smithi065.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-21T14:41:20.103 INFO:teuthology.orchestra.run.smithi065.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-21T14:41:20.104 INFO:teuthology.orchestra.run.smithi065.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-21T14:41:20.104 INFO:teuthology.orchestra.run.smithi065.stdout: thermald upower usbmuxd 2022-03-21T14:41:20.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-21T14:41:20.192 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2022-03-21T14:41:20.283 INFO:teuthology.orchestra.run.smithi065.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-21T14:41:20.284 INFO:teuthology.orchestra.run.smithi065.stdout:Need to get 80.1 MB of archives. 2022-03-21T14:41:20.284 INFO:teuthology.orchestra.run.smithi065.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-21T14:41:20.284 INFO:teuthology.orchestra.run.smithi065.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-21T14:41:20.370 INFO:teuthology.orchestra.run.smithi043.stdout:Building dependency tree... 2022-03-21T14:41:20.370 INFO:teuthology.orchestra.run.smithi043.stdout:Reading state information... 2022-03-21T14:41:20.434 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-21T14:41:20.515 INFO:teuthology.orchestra.run.smithi043.stdout:The following additional packages will be installed: 2022-03-21T14:41:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-21T14:41:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-21T14:41:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-21T14:41:20.517 INFO:teuthology.orchestra.run.smithi043.stdout:Suggested packages: 2022-03-21T14:41:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-21T14:41:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-21T14:41:20.539 INFO:teuthology.orchestra.run.smithi065.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-21T14:41:20.561 INFO:teuthology.orchestra.run.smithi065.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-21T14:41:20.562 INFO:teuthology.orchestra.run.smithi043.stdout:The following NEW packages will be installed: 2022-03-21T14:41:20.563 INFO:teuthology.orchestra.run.smithi043.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-21T14:41:20.563 INFO:teuthology.orchestra.run.smithi043.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-21T14:41:20.563 INFO:teuthology.orchestra.run.smithi043.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-21T14:41:20.563 INFO:teuthology.orchestra.run.smithi043.stdout: thermald upower usbmuxd 2022-03-21T14:41:20.564 INFO:teuthology.orchestra.run.smithi065.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-21T14:41:20.648 INFO:teuthology.orchestra.run.smithi065.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-21T14:41:20.650 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-21T14:41:20.668 INFO:teuthology.orchestra.run.smithi065.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-21T14:41:20.678 INFO:teuthology.orchestra.run.smithi043.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-21T14:41:20.678 INFO:teuthology.orchestra.run.smithi043.stdout:Need to get 80.1 MB of archives. 2022-03-21T14:41:20.679 INFO:teuthology.orchestra.run.smithi043.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-21T14:41:20.679 INFO:teuthology.orchestra.run.smithi043.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-21T14:41:20.856 INFO:teuthology.orchestra.run.smithi043.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-21T14:41:20.879 INFO:teuthology.orchestra.run.smithi043.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-21T14:41:20.879 INFO:teuthology.orchestra.run.smithi043.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-21T14:41:20.986 INFO:teuthology.orchestra.run.smithi043.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-21T14:41:20.987 INFO:teuthology.orchestra.run.smithi043.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-21T14:41:21.487 INFO:teuthology.orchestra.run.smithi065.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-21T14:41:21.687 INFO:teuthology.orchestra.run.smithi043.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-21T14:41:21.743 INFO:teuthology.orchestra.run.smithi065.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-21T14:41:21.864 INFO:teuthology.orchestra.run.smithi043.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-21T14:41:22.124 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-21T14:41:22.124 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-21T14:41:22.125 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-21T14:41:22.129 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-21T14:41:22.131 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-21T14:41:22.534 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 80.1 MB in 2s (36.5 MB/s) 2022-03-21T14:41:22.693 INFO:teuthology.orchestra.run.smithi043.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-21T14:41:22.694 INFO:teuthology.orchestra.run.smithi043.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-21T14:41:22.694 INFO:teuthology.orchestra.run.smithi043.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-21T14:41:22.697 INFO:teuthology.orchestra.run.smithi043.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-21T14:41:22.698 INFO:teuthology.orchestra.run.smithi043.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-21T14:41:22.734 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-21T14:41:23.115 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 80.1 MB in 2s (37.6 MB/s) 2022-03-21T14:41:23.342 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-21T14:41:23.466 INFO:teuthology.orchestra.run.smithi065.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-21T14:41:23.467 INFO:teuthology.orchestra.run.smithi065.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-21T14:41:23.468 INFO:teuthology.orchestra.run.smithi065.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-21T14:41:23.474 INFO:teuthology.orchestra.run.smithi065.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-21T14:41:23.475 INFO:teuthology.orchestra.run.smithi065.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-21T14:41:23.891 INFO:teuthology.orchestra.run.smithi065.stdout:Fetched 80.1 MB in 3s (23.8 MB/s) 2022-03-21T14:41:24.082 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-21T14:41:24.207 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-21T14:41:24.211 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-21T14:41:24.356 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-21T14:41:24.710 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-21T14:41:24.724 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-21T14:41:24.768 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-21T14:41:24.798 INFO:teuthology.orchestra.run.smithi043.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-21T14:41:24.803 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-21T14:41:24.922 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-21T14:41:25.104 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-21T14:41:25.117 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-21T14:41:25.162 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-21T14:41:25.208 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-21T14:41:25.221 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-21T14:41:25.250 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-21T14:41:25.514 INFO:teuthology.orchestra.run.smithi065.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-21T14:41:25.519 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-21T14:41:25.524 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-21T14:41:25.537 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-21T14:41:25.578 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-21T14:41:25.582 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-21T14:41:25.591 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-21T14:41:25.642 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-21T14:41:25.670 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-21T14:41:25.935 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-21T14:41:25.950 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-21T14:41:25.981 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-21T14:41:25.985 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-21T14:41:25.995 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-21T14:41:26.039 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-21T14:41:26.057 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-21T14:41:26.070 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-21T14:41:26.198 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-21T14:41:26.304 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-21T14:41:26.318 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-21T14:41:26.350 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-21T14:41:26.354 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:26.365 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-21T14:41:26.400 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-21T14:41:26.668 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-21T14:41:26.682 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-21T14:41:26.686 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-21T14:41:26.701 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-21T14:41:26.726 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-21T14:41:26.744 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:27.237 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-21T14:41:27.252 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-21T14:41:27.295 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-21T14:41:27.624 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-21T14:41:27.638 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-21T14:41:27.698 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-21T14:41:28.018 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-21T14:41:28.031 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-21T14:41:28.092 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:28.639 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-21T14:41:28.655 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-21T14:41:28.764 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:28.953 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-21T14:41:28.967 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-21T14:41:29.067 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:29.259 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-21T14:41:29.273 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-21T14:41:29.317 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:29.548 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-21T14:41:29.562 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-21T14:41:29.606 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:30.403 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-21T14:41:30.418 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-21T14:41:30.515 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:31.031 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-21T14:41:31.046 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-21T14:41:31.089 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:35.354 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-21T14:41:35.371 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-21T14:41:35.412 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-21T14:41:35.516 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-21T14:41:35.535 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-21T14:41:35.574 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-21T14:41:35.717 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-21T14:41:35.730 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-21T14:41:35.775 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-21T14:41:35.919 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-21T14:41:35.932 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-21T14:41:35.969 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-21T14:41:36.153 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package thermald. 2022-03-21T14:41:36.167 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-21T14:41:36.236 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-21T14:41:36.324 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package thermald. 2022-03-21T14:41:36.339 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-21T14:41:36.374 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-21T14:41:36.794 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package upower. 2022-03-21T14:41:36.807 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-21T14:41:36.854 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-21T14:41:36.896 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package upower. 2022-03-21T14:41:36.912 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-21T14:41:36.965 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-21T14:41:37.072 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-21T14:41:37.090 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-21T14:41:37.121 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-21T14:41:37.286 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package usbmuxd. 2022-03-21T14:41:37.299 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-21T14:41:37.336 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-21T14:41:37.361 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package usbmuxd. 2022-03-21T14:41:37.375 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-21T14:41:37.421 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-21T14:41:37.465 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-21T14:41:37.479 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-21T14:41:37.523 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-21T14:41:37.643 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-21T14:41:37.707 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-21T14:41:37.766 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-21T14:41:37.845 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-21T14:41:37.893 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package thermald. 2022-03-21T14:41:37.907 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-21T14:41:37.908 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-21T14:41:37.952 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-21T14:41:37.971 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-21T14:41:38.026 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-21T14:41:38.097 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-21T14:41:38.135 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-21T14:41:38.223 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-21T14:41:38.269 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-21T14:41:38.341 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-21T14:41:38.396 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-21T14:41:38.450 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-21T14:41:38.509 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package upower. 2022-03-21T14:41:38.525 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-21T14:41:38.578 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-21T14:41:38.853 INFO:teuthology.orchestra.run.smithi154.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-21T14:41:38.896 INFO:teuthology.orchestra.run.smithi043.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-21T14:41:38.906 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-21T14:41:38.944 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-21T14:41:39.002 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package usbmuxd. 2022-03-21T14:41:39.017 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-21T14:41:39.059 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-21T14:41:39.106 INFO:teuthology.orchestra.run.smithi154.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-21T14:41:39.114 INFO:teuthology.orchestra.run.smithi154.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-21T14:41:39.115 INFO:teuthology.orchestra.run.smithi154.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-21T14:41:39.115 INFO:teuthology.orchestra.run.smithi043.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-21T14:41:39.125 INFO:teuthology.orchestra.run.smithi043.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-21T14:41:39.126 INFO:teuthology.orchestra.run.smithi043.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-21T14:41:39.351 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-21T14:41:39.464 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-21T14:41:39.565 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-21T14:41:39.594 INFO:teuthology.orchestra.run.smithi043.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-21T14:41:39.656 INFO:teuthology.orchestra.run.smithi154.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-21T14:41:39.708 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-21T14:41:39.807 INFO:teuthology.orchestra.run.smithi043.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-21T14:41:39.843 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-21T14:41:39.909 INFO:teuthology.orchestra.run.smithi154.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-21T14:41:39.961 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-21T14:41:39.965 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-21T14:41:39.969 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-21T14:41:40.111 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-21T14:41:40.275 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-21T14:41:40.276 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-21T14:41:40.309 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-21T14:41:40.309 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-21T14:41:40.544 INFO:teuthology.orchestra.run.smithi065.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-21T14:41:40.688 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-21T14:41:40.753 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:40.790 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:40.875 INFO:teuthology.orchestra.run.smithi065.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-21T14:41:40.886 INFO:teuthology.orchestra.run.smithi065.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-21T14:41:40.886 INFO:teuthology.orchestra.run.smithi065.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-21T14:41:41.363 INFO:teuthology.orchestra.run.smithi065.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-21T14:41:41.592 INFO:teuthology.orchestra.run.smithi065.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-21T14:41:41.643 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-21T14:41:42.040 INFO:teuthology.orchestra.run.smithi043.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-21T14:41:42.040 INFO:teuthology.orchestra.run.smithi043.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-21T14:41:42.045 INFO:teuthology.orchestra.run.smithi154.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-21T14:41:42.045 INFO:teuthology.orchestra.run.smithi154.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-21T14:41:42.070 INFO:teuthology.orchestra.run.smithi065.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-21T14:41:42.071 INFO:teuthology.orchestra.run.smithi065.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-21T14:41:42.188 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:42.192 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:42.555 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:43.519 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:43.565 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:43.883 INFO:teuthology.orchestra.run.smithi065.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-21T14:41:43.883 INFO:teuthology.orchestra.run.smithi065.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-21T14:41:44.057 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:44.916 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-21T14:41:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-21T14:41:45.053 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-21T14:41:45.179 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-21T14:41:45.434 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:46.794 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-21T14:41:46.919 INFO:teuthology.orchestra.run.smithi065.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-21T14:41:49.051 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-21T14:41:49.365 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-21T14:41:49.712 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-21T14:41:50.200 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-21T14:41:50.300 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:50.490 INFO:teuthology.orchestra.run.smithi043.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-21T14:41:50.490 INFO:teuthology.orchestra.run.smithi043.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-21T14:41:50.596 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:51.245 INFO:teuthology.orchestra.run.smithi065.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-21T14:41:51.299 INFO:teuthology.orchestra.run.smithi154.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-21T14:41:51.299 INFO:teuthology.orchestra.run.smithi154.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-21T14:41:51.903 INFO:teuthology.orchestra.run.smithi065.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-21T14:41:52.011 INFO:teuthology.orchestra.run.smithi065.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-21T14:41:52.205 INFO:teuthology.orchestra.run.smithi065.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-21T14:41:52.206 INFO:teuthology.orchestra.run.smithi065.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-21T14:42:14.287 INFO:teuthology.orchestra.run.smithi043.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-21T14:42:14.288 INFO:teuthology.orchestra.run.smithi043.stdout:Sourcing file `/etc/default/grub' 2022-03-21T14:42:14.298 INFO:teuthology.orchestra.run.smithi043.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-21T14:42:14.374 INFO:teuthology.orchestra.run.smithi043.stdout:Generating grub configuration file ... 2022-03-21T14:42:14.729 INFO:teuthology.orchestra.run.smithi043.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-21T14:42:14.746 INFO:teuthology.orchestra.run.smithi154.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-21T14:42:14.747 INFO:teuthology.orchestra.run.smithi154.stdout:Sourcing file `/etc/default/grub' 2022-03-21T14:42:14.749 INFO:teuthology.orchestra.run.smithi043.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-21T14:42:14.757 INFO:teuthology.orchestra.run.smithi154.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-21T14:42:14.774 INFO:teuthology.orchestra.run.smithi154.stdout:Generating grub configuration file ... 2022-03-21T14:42:15.106 INFO:teuthology.orchestra.run.smithi154.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-21T14:42:15.132 INFO:teuthology.orchestra.run.smithi154.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-21T14:42:15.178 INFO:teuthology.orchestra.run.smithi043.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-21T14:42:15.189 INFO:teuthology.orchestra.run.smithi043.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-21T14:42:15.519 INFO:teuthology.orchestra.run.smithi154.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-21T14:42:15.530 INFO:teuthology.orchestra.run.smithi154.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-21T14:42:16.200 INFO:teuthology.orchestra.run.smithi065.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-21T14:42:16.201 INFO:teuthology.orchestra.run.smithi065.stdout:Sourcing file `/etc/default/grub' 2022-03-21T14:42:16.210 INFO:teuthology.orchestra.run.smithi065.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-21T14:42:16.228 INFO:teuthology.orchestra.run.smithi065.stdout:Generating grub configuration file ... 2022-03-21T14:42:16.551 INFO:teuthology.orchestra.run.smithi065.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-21T14:42:16.579 INFO:teuthology.orchestra.run.smithi065.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-21T14:42:16.968 INFO:teuthology.orchestra.run.smithi065.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-21T14:42:16.980 INFO:teuthology.orchestra.run.smithi065.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-21T14:42:17.815 INFO:teuthology.orchestra.run.smithi043.stdout:done 2022-03-21T14:42:18.159 DEBUG:teuthology.orchestra.run.smithi043:> dpkg -s linux-image-generic-hwe-20.04 2022-03-21T14:42:18.188 INFO:teuthology.orchestra.run.smithi043.stdout:Package: linux-image-generic-hwe-20.04 2022-03-21T14:42:18.188 INFO:teuthology.orchestra.run.smithi043.stdout:Status: install ok installed 2022-03-21T14:42:18.188 INFO:teuthology.orchestra.run.smithi043.stdout:Priority: optional 2022-03-21T14:42:18.188 INFO:teuthology.orchestra.run.smithi043.stdout:Section: kernel 2022-03-21T14:42:18.189 INFO:teuthology.orchestra.run.smithi043.stdout:Installed-Size: 19 2022-03-21T14:42:18.189 INFO:teuthology.orchestra.run.smithi043.stdout:Maintainer: Ubuntu Kernel Team 2022-03-21T14:42:18.189 INFO:teuthology.orchestra.run.smithi043.stdout:Architecture: amd64 2022-03-21T14:42:18.189 INFO:teuthology.orchestra.run.smithi043.stdout:Source: linux-meta-hwe-5.13 2022-03-21T14:42:18.189 INFO:teuthology.orchestra.run.smithi043.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-21T14:42:18.190 INFO:teuthology.orchestra.run.smithi043.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-21T14:42:18.190 INFO:teuthology.orchestra.run.smithi043.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-21T14:42:18.190 INFO:teuthology.orchestra.run.smithi043.stdout:Recommends: thermald 2022-03-21T14:42:18.190 INFO:teuthology.orchestra.run.smithi043.stdout:Description: Generic Linux kernel image 2022-03-21T14:42:18.191 INFO:teuthology.orchestra.run.smithi043.stdout: This package will always depend on the latest generic kernel image 2022-03-21T14:42:18.191 INFO:teuthology.orchestra.run.smithi043.stdout: available. 2022-03-21T14:42:18.191 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-21T14:42:18.191 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-21T14:42:18.192 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-21T14:42:18.192 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-21T14:42:18.192 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi043.front.sepia.ceph.com, path=None, version=distro) 2022-03-21T14:42:18.192 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-get clean 2022-03-21T14:42:18.250 INFO:teuthology.orchestra.run.smithi154.stdout:done 2022-03-21T14:42:18.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-get update 2022-03-21T14:42:18.448 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:42:18.458 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-21T14:42:18.474 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-21T14:42:18.478 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-21T14:42:18.640 DEBUG:teuthology.orchestra.run.smithi154:> dpkg -s linux-image-generic-hwe-20.04 2022-03-21T14:42:18.668 INFO:teuthology.orchestra.run.smithi154.stdout:Package: linux-image-generic-hwe-20.04 2022-03-21T14:42:18.669 INFO:teuthology.orchestra.run.smithi154.stdout:Status: install ok installed 2022-03-21T14:42:18.669 INFO:teuthology.orchestra.run.smithi154.stdout:Priority: optional 2022-03-21T14:42:18.669 INFO:teuthology.orchestra.run.smithi154.stdout:Section: kernel 2022-03-21T14:42:18.669 INFO:teuthology.orchestra.run.smithi154.stdout:Installed-Size: 19 2022-03-21T14:42:18.669 INFO:teuthology.orchestra.run.smithi154.stdout:Maintainer: Ubuntu Kernel Team 2022-03-21T14:42:18.670 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: amd64 2022-03-21T14:42:18.670 INFO:teuthology.orchestra.run.smithi154.stdout:Source: linux-meta-hwe-5.13 2022-03-21T14:42:18.670 INFO:teuthology.orchestra.run.smithi154.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-21T14:42:18.670 INFO:teuthology.orchestra.run.smithi154.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-21T14:42:18.671 INFO:teuthology.orchestra.run.smithi154.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-21T14:42:18.671 INFO:teuthology.orchestra.run.smithi154.stdout:Recommends: thermald 2022-03-21T14:42:18.671 INFO:teuthology.orchestra.run.smithi154.stdout:Description: Generic Linux kernel image 2022-03-21T14:42:18.671 INFO:teuthology.orchestra.run.smithi154.stdout: This package will always depend on the latest generic kernel image 2022-03-21T14:42:18.671 INFO:teuthology.orchestra.run.smithi154.stdout: available. 2022-03-21T14:42:18.672 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-21T14:42:18.672 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-21T14:42:18.672 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-21T14:42:18.672 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-21T14:42:18.672 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2022-03-21T14:42:18.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get clean 2022-03-21T14:42:18.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get update 2022-03-21T14:42:19.060 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-21T14:42:19.060 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:42:19.133 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-21T14:42:19.230 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-21T14:42:19.654 INFO:teuthology.orchestra.run.smithi065.stdout:done 2022-03-21T14:42:20.013 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2022-03-21T14:42:20.031 DEBUG:teuthology.orchestra.run.smithi065:> dpkg -s linux-image-generic-hwe-20.04 2022-03-21T14:42:20.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-21T14:42:20.054 INFO:teuthology.orchestra.run.smithi065.stdout:Package: linux-image-generic-hwe-20.04 2022-03-21T14:42:20.054 INFO:teuthology.orchestra.run.smithi065.stdout:Status: install ok installed 2022-03-21T14:42:20.054 INFO:teuthology.orchestra.run.smithi065.stdout:Priority: optional 2022-03-21T14:42:20.055 INFO:teuthology.orchestra.run.smithi065.stdout:Section: kernel 2022-03-21T14:42:20.055 INFO:teuthology.orchestra.run.smithi065.stdout:Installed-Size: 19 2022-03-21T14:42:20.055 INFO:teuthology.orchestra.run.smithi065.stdout:Maintainer: Ubuntu Kernel Team 2022-03-21T14:42:20.055 INFO:teuthology.orchestra.run.smithi065.stdout:Architecture: amd64 2022-03-21T14:42:20.056 INFO:teuthology.orchestra.run.smithi065.stdout:Source: linux-meta-hwe-5.13 2022-03-21T14:42:20.056 INFO:teuthology.orchestra.run.smithi065.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-21T14:42:20.056 INFO:teuthology.orchestra.run.smithi065.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-21T14:42:20.056 INFO:teuthology.orchestra.run.smithi065.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-21T14:42:20.056 INFO:teuthology.orchestra.run.smithi065.stdout:Recommends: thermald 2022-03-21T14:42:20.057 INFO:teuthology.orchestra.run.smithi065.stdout:Description: Generic Linux kernel image 2022-03-21T14:42:20.057 INFO:teuthology.orchestra.run.smithi065.stdout: This package will always depend on the latest generic kernel image 2022-03-21T14:42:20.057 INFO:teuthology.orchestra.run.smithi065.stdout: available. 2022-03-21T14:42:20.058 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-21T14:42:20.058 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-21T14:42:20.059 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-21T14:42:20.059 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-21T14:42:20.059 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi065.front.sepia.ceph.com, path=None, version=distro) 2022-03-21T14:42:20.060 DEBUG:teuthology.orchestra.run.smithi065:> sudo apt-get clean 2022-03-21T14:42:20.103 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2022-03-21T14:42:20.140 DEBUG:teuthology.orchestra.run.smithi065:> sudo apt-get update 2022-03-21T14:42:20.262 INFO:teuthology.orchestra.run.smithi043.stdout:Building dependency tree... 2022-03-21T14:42:20.263 INFO:teuthology.orchestra.run.smithi043.stdout:Reading state information... 2022-03-21T14:42:20.284 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:42:20.293 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-21T14:42:20.315 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-21T14:42:20.392 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-21T14:42:20.438 INFO:teuthology.orchestra.run.smithi043.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-21T14:42:20.439 INFO:teuthology.orchestra.run.smithi043.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-21T14:42:20.441 DEBUG:teuthology.orchestra.run.smithi043:> dpkg -s linux-image-generic-hwe-20.04 2022-03-21T14:42:20.463 INFO:teuthology.orchestra.run.smithi043.stdout:Package: linux-image-generic-hwe-20.04 2022-03-21T14:42:20.463 INFO:teuthology.orchestra.run.smithi043.stdout:Status: install ok installed 2022-03-21T14:42:20.463 INFO:teuthology.orchestra.run.smithi043.stdout:Priority: optional 2022-03-21T14:42:20.464 INFO:teuthology.orchestra.run.smithi043.stdout:Section: kernel 2022-03-21T14:42:20.464 INFO:teuthology.orchestra.run.smithi043.stdout:Installed-Size: 19 2022-03-21T14:42:20.464 INFO:teuthology.orchestra.run.smithi043.stdout:Maintainer: Ubuntu Kernel Team 2022-03-21T14:42:20.464 INFO:teuthology.orchestra.run.smithi043.stdout:Architecture: amd64 2022-03-21T14:42:20.465 INFO:teuthology.orchestra.run.smithi043.stdout:Source: linux-meta-hwe-5.13 2022-03-21T14:42:20.465 INFO:teuthology.orchestra.run.smithi043.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-21T14:42:20.465 INFO:teuthology.orchestra.run.smithi043.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-21T14:42:20.465 INFO:teuthology.orchestra.run.smithi043.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-21T14:42:20.465 INFO:teuthology.orchestra.run.smithi043.stdout:Recommends: thermald 2022-03-21T14:42:20.466 INFO:teuthology.orchestra.run.smithi043.stdout:Description: Generic Linux kernel image 2022-03-21T14:42:20.466 INFO:teuthology.orchestra.run.smithi043.stdout: This package will always depend on the latest generic kernel image 2022-03-21T14:42:20.466 INFO:teuthology.orchestra.run.smithi043.stdout: available. 2022-03-21T14:42:20.467 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2022-03-21T14:42:20.471 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.cov3WLe7v2 2022-03-21T14:42:20.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo cp /boot/grub/grub.cfg /tmp/tmp.cov3WLe7v2 2022-03-21T14:42:20.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.cov3WLe7v2 2022-03-21T14:42:20.567 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.cov3WLe7v2 is 8KB 2022-03-21T14:42:20.582 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.cov3WLe7v2 2022-03-21T14:42:20.588 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-21T14:42:20.643 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2022-03-21T14:42:20.645 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:42:20.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-21T14:42:20.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-21T14:42:20.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-21T14:42:20.672 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-21T14:42:20.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo update-grub 2022-03-21T14:42:20.731 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2022-03-21T14:42:20.906 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2022-03-21T14:42:20.906 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2022-03-21T14:42:21.089 INFO:teuthology.orchestra.run.smithi154.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-21T14:42:21.089 INFO:teuthology.orchestra.run.smithi154.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-21T14:42:21.091 DEBUG:teuthology.orchestra.run.smithi154:> dpkg -s linux-image-generic-hwe-20.04 2022-03-21T14:42:21.113 INFO:teuthology.orchestra.run.smithi154.stdout:Package: linux-image-generic-hwe-20.04 2022-03-21T14:42:21.114 INFO:teuthology.orchestra.run.smithi154.stdout:Status: install ok installed 2022-03-21T14:42:21.114 INFO:teuthology.orchestra.run.smithi154.stdout:Priority: optional 2022-03-21T14:42:21.114 INFO:teuthology.orchestra.run.smithi154.stdout:Section: kernel 2022-03-21T14:42:21.114 INFO:teuthology.orchestra.run.smithi154.stdout:Installed-Size: 19 2022-03-21T14:42:21.115 INFO:teuthology.orchestra.run.smithi154.stdout:Maintainer: Ubuntu Kernel Team 2022-03-21T14:42:21.115 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: amd64 2022-03-21T14:42:21.115 INFO:teuthology.orchestra.run.smithi154.stdout:Source: linux-meta-hwe-5.13 2022-03-21T14:42:21.115 INFO:teuthology.orchestra.run.smithi154.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-21T14:42:21.115 INFO:teuthology.orchestra.run.smithi154.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-21T14:42:21.116 INFO:teuthology.orchestra.run.smithi154.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-21T14:42:21.116 INFO:teuthology.orchestra.run.smithi154.stdout:Recommends: thermald 2022-03-21T14:42:21.116 INFO:teuthology.orchestra.run.smithi154.stdout:Description: Generic Linux kernel image 2022-03-21T14:42:21.116 INFO:teuthology.orchestra.run.smithi154.stdout: This package will always depend on the latest generic kernel image 2022-03-21T14:42:21.117 INFO:teuthology.orchestra.run.smithi154.stdout: available. 2022-03-21T14:42:21.117 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2022-03-21T14:42:21.121 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.k5Wtm1xznN 2022-03-21T14:42:21.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub/grub.cfg /tmp/tmp.k5Wtm1xznN 2022-03-21T14:42:21.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.k5Wtm1xznN 2022-03-21T14:42:21.170 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.k5Wtm1xznN is 8KB 2022-03-21T14:42:21.184 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.k5Wtm1xznN 2022-03-21T14:42:21.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-21T14:42:21.246 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:42:21.246 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-21T14:42:21.246 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-21T14:42:21.275 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-21T14:42:21.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo update-grub 2022-03-21T14:42:21.719 INFO:teuthology.orchestra.run.smithi043.stderr:Sourcing file `/etc/default/grub' 2022-03-21T14:42:21.722 INFO:teuthology.orchestra.run.smithi043.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-21T14:42:21.739 INFO:teuthology.orchestra.run.smithi043.stderr:Generating grub configuration file ... 2022-03-21T14:42:21.890 INFO:teuthology.orchestra.run.smithi065.stdout:Reading package lists... 2022-03-21T14:42:21.914 DEBUG:teuthology.orchestra.run.smithi065:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-21T14:42:21.981 INFO:teuthology.orchestra.run.smithi065.stdout:Reading package lists... 2022-03-21T14:42:22.054 INFO:teuthology.orchestra.run.smithi043.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-21T14:42:22.066 INFO:teuthology.orchestra.run.smithi043.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-21T14:42:22.140 INFO:teuthology.orchestra.run.smithi065.stdout:Building dependency tree... 2022-03-21T14:42:22.141 INFO:teuthology.orchestra.run.smithi065.stdout:Reading state information... 2022-03-21T14:42:22.319 INFO:teuthology.orchestra.run.smithi065.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-21T14:42:22.319 INFO:teuthology.orchestra.run.smithi065.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-21T14:42:22.321 DEBUG:teuthology.orchestra.run.smithi065:> dpkg -s linux-image-generic-hwe-20.04 2022-03-21T14:42:22.342 INFO:teuthology.orchestra.run.smithi065.stdout:Package: linux-image-generic-hwe-20.04 2022-03-21T14:42:22.342 INFO:teuthology.orchestra.run.smithi065.stdout:Status: install ok installed 2022-03-21T14:42:22.342 INFO:teuthology.orchestra.run.smithi065.stdout:Priority: optional 2022-03-21T14:42:22.342 INFO:teuthology.orchestra.run.smithi065.stdout:Section: kernel 2022-03-21T14:42:22.343 INFO:teuthology.orchestra.run.smithi065.stdout:Installed-Size: 19 2022-03-21T14:42:22.343 INFO:teuthology.orchestra.run.smithi065.stdout:Maintainer: Ubuntu Kernel Team 2022-03-21T14:42:22.343 INFO:teuthology.orchestra.run.smithi065.stdout:Architecture: amd64 2022-03-21T14:42:22.343 INFO:teuthology.orchestra.run.smithi065.stdout:Source: linux-meta-hwe-5.13 2022-03-21T14:42:22.344 INFO:teuthology.orchestra.run.smithi065.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-21T14:42:22.344 INFO:teuthology.orchestra.run.smithi065.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-21T14:42:22.344 INFO:teuthology.orchestra.run.smithi065.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-21T14:42:22.344 INFO:teuthology.orchestra.run.smithi065.stdout:Recommends: thermald 2022-03-21T14:42:22.344 INFO:teuthology.orchestra.run.smithi065.stdout:Description: Generic Linux kernel image 2022-03-21T14:42:22.345 INFO:teuthology.orchestra.run.smithi065.stdout: This package will always depend on the latest generic kernel image 2022-03-21T14:42:22.345 INFO:teuthology.orchestra.run.smithi065.stdout: available. 2022-03-21T14:42:22.345 DEBUG:teuthology.orchestra.run.smithi065:> mktemp 2022-03-21T14:42:22.350 INFO:teuthology.orchestra.run.smithi065.stdout:/tmp/tmp.FcIhJgUfAZ 2022-03-21T14:42:22.351 DEBUG:teuthology.orchestra.run.smithi065:> sudo cp /boot/grub/grub.cfg /tmp/tmp.FcIhJgUfAZ 2022-03-21T14:42:22.406 DEBUG:teuthology.orchestra.run.smithi065:> sudo chmod 0666 /tmp/tmp.FcIhJgUfAZ 2022-03-21T14:42:22.442 DEBUG:teuthology.orchestra.remote:smithi065:/tmp/tmp.FcIhJgUfAZ is 8KB 2022-03-21T14:42:22.456 DEBUG:teuthology.orchestra.run.smithi065:> rm -fr /tmp/tmp.FcIhJgUfAZ 2022-03-21T14:42:22.462 DEBUG:teuthology.orchestra.run.smithi065:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-21T14:42:22.473 INFO:teuthology.orchestra.run.smithi154.stderr:Sourcing file `/etc/default/grub' 2022-03-21T14:42:22.475 INFO:teuthology.orchestra.run.smithi154.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-21T14:42:22.507 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2022-03-21T14:42:22.515 INFO:teuthology.orchestra.run.smithi043.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-21T14:42:22.519 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:42:22.520 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-21T14:42:22.520 DEBUG:teuthology.orchestra.run.smithi065:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-21T14:42:22.527 INFO:teuthology.orchestra.run.smithi043.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-21T14:42:22.558 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-21T14:42:22.558 DEBUG:teuthology.orchestra.run.smithi065:> sudo update-grub 2022-03-21T14:42:22.752 INFO:teuthology.orchestra.run.smithi154.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-21T14:42:22.763 INFO:teuthology.orchestra.run.smithi154.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-21T14:42:23.148 INFO:teuthology.orchestra.run.smithi154.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-21T14:42:23.160 INFO:teuthology.orchestra.run.smithi154.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-21T14:42:23.791 INFO:teuthology.orchestra.run.smithi065.stderr:Sourcing file `/etc/default/grub' 2022-03-21T14:42:23.793 INFO:teuthology.orchestra.run.smithi065.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-21T14:42:23.802 INFO:teuthology.orchestra.run.smithi065.stderr:Generating grub configuration file ... 2022-03-21T14:42:24.062 INFO:teuthology.orchestra.run.smithi065.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-21T14:42:24.073 INFO:teuthology.orchestra.run.smithi065.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-21T14:42:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:done 2022-03-21T14:42:24.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo shutdown -r now 2022-03-21T14:42:24.414 INFO:teuthology.orchestra.run.smithi065.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-21T14:42:24.426 INFO:teuthology.orchestra.run.smithi065.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-21T14:42:24.787 INFO:teuthology.orchestra.run.smithi154.stderr:done 2022-03-21T14:42:24.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2022-03-21T14:42:26.108 INFO:teuthology.orchestra.run.smithi065.stderr:done 2022-03-21T14:42:26.110 DEBUG:teuthology.orchestra.run.smithi065:> sudo shutdown -r now 2022-03-21T14:42:54.238 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-21T14:42:54.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2022-03-21T14:42:54.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:42:54.793 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-21T14:42:54.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2022-03-21T14:42:54.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:42:56.115 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-21T14:42:56.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com' 2022-03-21T14:42:56.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi065.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:43:12.715 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.43 2022-03-21T14:43:13.227 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2022-03-21T14:43:14.507 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.65 2022-03-21T14:43:42.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2022-03-21T14:43:42.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:43:43.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2022-03-21T14:43:43.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:43:44.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com' 2022-03-21T14:43:44.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi065.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:43:58.238 DEBUG:teuthology.orchestra.run.smithi043:> true 2022-03-21T14:43:58.788 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2022-03-21T14:43:58.788 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-21T14:43:58.789 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2022-03-21T14:43:58.796 INFO:teuthology.orchestra.run.smithi043.stdout:5.13.0-35-generic 2022-03-21T14:43:58.796 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-21T14:43:58.796 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-21T14:43:58.797 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-21T14:43:59.797 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-21T14:43:59.798 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-21T14:43:59.870 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2022-03-21T14:43:59.885 DEBUG:teuthology.parallel:result is None 2022-03-21T14:44:01.610 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2022-03-21T14:44:02.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.65 2022-03-21T14:44:31.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2022-03-21T14:44:31.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:44:31.895 DEBUG:teuthology.orchestra.run.smithi154:> true 2022-03-21T14:44:32.454 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2022-03-21T14:44:32.454 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-35-generic"... 2022-03-21T14:44:32.455 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2022-03-21T14:44:32.460 INFO:teuthology.orchestra.run.smithi154.stdout:5.13.0-35-generic 2022-03-21T14:44:32.461 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-21T14:44:32.461 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-21T14:44:32.461 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-21T14:44:32.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com' 2022-03-21T14:44:32.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi065.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:44:33.070 DEBUG:teuthology.orchestra.run.smithi065:> true 2022-03-21T14:44:33.462 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-21T14:44:33.463 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-21T14:44:33.534 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2022-03-21T14:44:33.560 DEBUG:teuthology.parallel:result is None 2022-03-21T14:44:33.683 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi065.front.sepia.ceph.com' 2022-03-21T14:44:33.683 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-35-generic"... 2022-03-21T14:44:33.683 DEBUG:teuthology.orchestra.run.smithi065:> uname -r 2022-03-21T14:44:33.688 INFO:teuthology.orchestra.run.smithi065.stdout:5.13.0-35-generic 2022-03-21T14:44:33.689 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-21T14:44:33.689 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-21T14:44:33.689 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-21T14:44:34.690 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-21T14:44:34.691 DEBUG:teuthology.orchestra.run.smithi065:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-21T14:44:34.762 INFO:teuthology.orchestra.run.smithi065.stdout:ttyS1 2022-03-21T14:44:34.792 DEBUG:teuthology.parallel:result is None 2022-03-21T14:44:34.793 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-21T14:44:34.803 INFO:teuthology.task.internal:Creating test directory... 2022-03-21T14:44:34.803 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-21T14:44:34.806 DEBUG:teuthology.orchestra.run.smithi065:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-21T14:44:34.809 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-21T14:44:34.816 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-21T14:44:34.846 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-21T14:44:34.940 INFO:teuthology.task.internal:Creating archive directory... 2022-03-21T14:44:34.940 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-21T14:44:34.943 DEBUG:teuthology.orchestra.run.smithi065:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-21T14:44:34.946 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-21T14:44:35.026 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-21T14:44:35.036 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-21T14:44:35.037 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-21T14:44:35.039 DEBUG:teuthology.orchestra.run.smithi065:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-21T14:44:35.042 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-21T14:44:35.066 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T14:44:35.068 INFO:teuthology.orchestra.run.smithi065.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T14:44:35.072 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T14:44:35.073 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T14:44:35.079 INFO:teuthology.orchestra.run.smithi065.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T14:44:35.083 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-21T14:44:35.085 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-21T14:44:35.096 INFO:teuthology.task.internal:Configuring sudo... 2022-03-21T14:44:35.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-21T14:44:35.099 DEBUG:teuthology.orchestra.run.smithi065:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-21T14:44:35.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-21T14:44:35.134 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-21T14:44:35.177 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-21T14:44:35.178 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-21T14:44:35.181 DEBUG:teuthology.orchestra.run.smithi065:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-21T14:44:35.183 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-21T14:44:35.191 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-21T14:44:35.197 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-21T14:44:35.203 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:44:35.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-21T14:44:35.268 DEBUG:teuthology.orchestra.run.smithi065:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-21T14:44:35.276 DEBUG:teuthology.orchestra.run.smithi065:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-21T14:44:35.282 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:44:35.283 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-21T14:44:35.306 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-21T14:44:35.313 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-21T14:44:35.365 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:44:35.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-21T14:44:35.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2022-03-21T14:44:35.394 DEBUG:teuthology.orchestra.run.smithi065:> sudo service rsyslog restart 2022-03-21T14:44:35.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2022-03-21T14:44:35.445 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-21T14:44:35.456 INFO:teuthology.task.internal:Starting timer... 2022-03-21T14:44:35.456 INFO:teuthology.run_tasks:Running task pcp... 2022-03-21T14:44:35.520 INFO:teuthology.run_tasks:Running task selinux... 2022-03-21T14:44:35.570 DEBUG:teuthology.task.selinux:Excluding smithi043: OS 'ubuntu' does not support SELinux 2022-03-21T14:44:35.571 DEBUG:teuthology.task.selinux:Excluding smithi065: OS 'ubuntu' does not support SELinux 2022-03-21T14:44:35.571 DEBUG:teuthology.task.selinux:Excluding smithi154: OS 'ubuntu' does not support SELinux 2022-03-21T14:44:35.572 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-21T14:44:35.572 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-21T14:44:35.573 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-21T14:44:35.573 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-21T14:44:35.631 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-03-21T14:44:35.637 INFO:teuthology.repo_utils:Fetching master from origin 2022-03-21T14:44:35.671 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-21T14:44:35.698 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-21T14:44:35.700 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi065.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-21T14:47:53.623 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi065.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2022-03-21T14:47:53.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2022-03-21T14:47:53.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:47:53.674 DEBUG:teuthology.orchestra.run.smithi043:> true 2022-03-21T14:47:53.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com' 2022-03-21T14:47:53.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi065.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:47:53.848 DEBUG:teuthology.orchestra.run.smithi065:> true 2022-03-21T14:47:53.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2022-03-21T14:47:53.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-21T14:47:54.019 DEBUG:teuthology.orchestra.run.smithi154:> true 2022-03-21T14:47:54.136 INFO:teuthology.run_tasks:Running task clock... 2022-03-21T14:47:54.147 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-21T14:47:54.147 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-21T14:47:54.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-21T14:47:54.152 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-21T14:47:54.152 DEBUG:teuthology.orchestra.run.smithi065:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-21T14:47:54.155 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-21T14:47:54.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-21T14:47:54.183 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-21T14:47:54.184 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: Command line: ntpd -gq 2022-03-21T14:47:54.184 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: proto: precision = 0.074 usec (-24) 2022-03-21T14:47:54.185 INFO:teuthology.orchestra.run.smithi043.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-21T14:47:54.185 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-21T14:47:54.186 INFO:teuthology.orchestra.run.smithi043.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-21T14:47:54.187 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: restrict ::: KOD does nothing without LIMITED. 2022-03-21T14:47:54.187 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: Listen and drop on 0 v6wildcard [::]:123 2022-03-21T14:47:54.188 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-21T14:47:54.188 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: Listen normally on 2 lo 127.0.0.1:123 2022-03-21T14:47:54.188 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: Listen normally on 3 ens1f0 172.21.15.43:123 2022-03-21T14:47:54.188 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: Listen normally on 4 lo [::1]:123 2022-03-21T14:47:54.189 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1358%4]:123 2022-03-21T14:47:54.189 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:47:54 ntpd[9431]: Listening on routing socket on fd #22 for interface updates 2022-03-21T14:47:54.190 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-21T14:47:54.190 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: Command line: ntpd -gq 2022-03-21T14:47:54.191 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: proto: precision = 0.093 usec (-23) 2022-03-21T14:47:54.191 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-21T14:47:54.191 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: Command line: ntpd -gq 2022-03-21T14:47:54.192 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: proto: precision = 0.092 usec (-23) 2022-03-21T14:47:54.192 INFO:teuthology.orchestra.run.smithi154.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-21T14:47:54.192 INFO:teuthology.orchestra.run.smithi065.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-21T14:47:54.193 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-21T14:47:54.194 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-21T14:47:54.194 INFO:teuthology.orchestra.run.smithi154.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-21T14:47:54.194 INFO:teuthology.orchestra.run.smithi065.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-21T14:47:54.196 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: restrict ::: KOD does nothing without LIMITED. 2022-03-21T14:47:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: Listen and drop on 0 v6wildcard [::]:123 2022-03-21T14:47:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-21T14:47:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: Listen normally on 2 lo 127.0.0.1:123 2022-03-21T14:47:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: Listen normally on 3 enp3s0f1 172.21.15.154:123 2022-03-21T14:47:54.198 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: Listen normally on 4 lo [::1]:123 2022-03-21T14:47:54.198 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7761%5]:123 2022-03-21T14:47:54.198 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:47:54 ntpd[9563]: Listening on routing socket on fd #22 for interface updates 2022-03-21T14:47:54.198 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: restrict ::: KOD does nothing without LIMITED. 2022-03-21T14:47:54.199 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: Listen and drop on 0 v6wildcard [::]:123 2022-03-21T14:47:54.199 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-21T14:47:54.199 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: Listen normally on 2 lo 127.0.0.1:123 2022-03-21T14:47:54.199 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: Listen normally on 3 enp3s0f1 172.21.15.65:123 2022-03-21T14:47:54.199 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: Listen normally on 4 lo [::1]:123 2022-03-21T14:47:54.200 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f63d%5]:123 2022-03-21T14:47:54.200 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:47:54 ntpd[9549]: Listening on routing socket on fd #22 for interface updates 2022-03-21T14:47:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:55 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:55 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:55.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:55 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:55.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:55 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:55.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:55 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:55.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:55 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:55.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:55.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:55.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:55.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:55.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:55.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:55.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:55.192 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:55.192 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:55.192 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:55.192 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:55.193 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:55.193 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:55.193 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:55.193 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:55.193 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:55.194 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:55.194 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:55 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:56 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:56 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:56 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:56 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:56 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:56 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:56.189 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:56 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:56.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:56 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:56.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:56 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:56.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:56 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:56.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:56 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:56.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:56 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:56.191 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:56 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:56.191 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:56 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:56.191 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:56 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:56.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:56 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:56.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:56 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:56.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:56 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:57 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:57 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:57 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:57 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:57 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:57 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:57.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:57.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:57.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:57.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:57.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:57.192 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:57.192 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:57.192 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:57.192 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:57.192 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:57.193 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:57.193 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:57.193 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:57.193 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:57.194 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:57.194 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:57.194 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:57.194 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:57 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:58.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:58.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:58.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:58.188 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:58 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:58.189 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:58 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:58 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:58 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:58 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:58 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:58.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:58 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:58.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:58.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:58.194 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:58.194 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:58.194 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:58.194 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:58 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:47:59 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:59.191 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.191 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:59.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:59.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:59.194 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:47:59 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:59.195 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.195 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.195 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.195 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.196 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.196 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.196 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.196 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.197 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:59.197 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:59.197 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.197 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.198 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:59.198 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:47:59.198 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:47:59.198 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:47:59.198 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:47:59.199 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:47:59 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:00 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:00.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:00 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.190 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:00 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:00 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:00 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:00 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:00.191 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:00 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:00.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.193 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.194 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:00.194 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:00.194 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:00.194 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:00.195 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:00.195 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:00 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:01.185 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:01 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:01 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:01 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:01 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:01 ntpd[9431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:01 ntpd[9431]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:01 ntpd[9431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:01 ntpd[9431]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:21 Mar 14:48:01 ntpd[9431]: ntpd: time slew -0.002068 s 2022-03-21T14:48:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:ntpd: time slew -0.002068s 2022-03-21T14:48:01.740 INFO:teuthology.orchestra.run.smithi043.stderr:21 Mar 14:48:01 ntpd[9431]: can't open /var/log/ntpstats/loopstats.20220321: Permission denied 2022-03-21T14:48:01.741 INFO:teuthology.orchestra.run.smithi043.stdout: remote refid st t when poll reach delay offset jitter 2022-03-21T14:48:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================== 2022-03-21T14:48:01.741 INFO:teuthology.orchestra.run.smithi043.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:01.742 INFO:teuthology.orchestra.run.smithi043.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:01.742 INFO:teuthology.orchestra.run.smithi043.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:01.742 INFO:teuthology.orchestra.run.smithi043.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:01.745 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.745 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.746 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.746 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.746 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.746 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.747 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.747 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.747 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:01.747 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:01.748 INFO:teuthology.orchestra.run.smithi065.stdout:21 Mar 14:48:01 ntpd[9549]: ntpd: time slew +0.000280 s 2022-03-21T14:48:01.748 INFO:teuthology.orchestra.run.smithi065.stdout:ntpd: time slew +0.000280s 2022-03-21T14:48:01.749 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.749 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.749 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.750 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.750 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.750 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.750 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:01.751 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:01.751 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:01.751 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:01.751 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:01.752 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:01 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:01.752 INFO:teuthology.orchestra.run.smithi065.stderr:21 Mar 14:48:01 ntpd[9549]: can't open /var/log/ntpstats/loopstats.20220321: Permission denied 2022-03-21T14:48:01.753 INFO:teuthology.orchestra.run.smithi065.stdout: remote refid st t when poll reach delay offset jitter 2022-03-21T14:48:01.753 INFO:teuthology.orchestra.run.smithi065.stdout:============================================================================== 2022-03-21T14:48:01.754 INFO:teuthology.orchestra.run.smithi065.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:01.754 INFO:teuthology.orchestra.run.smithi065.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:01.754 INFO:teuthology.orchestra.run.smithi065.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:01.754 INFO:teuthology.orchestra.run.smithi065.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:02.190 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:02 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:02.190 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:02 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:02.190 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:02 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:02.190 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:02 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:02.191 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:02 ntpd[9563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-21T14:48:02.191 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:02 ntpd[9563]: can't open /var/log/ntpstats/rawstats.20220321: Permission denied 2022-03-21T14:48:02.191 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:02 ntpd[9563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-21T14:48:02.191 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:02 ntpd[9563]: can't open /var/log/ntpstats/peerstats.20220321: Permission denied 2022-03-21T14:48:02.192 INFO:teuthology.orchestra.run.smithi154.stdout:21 Mar 14:48:02 ntpd[9563]: ntpd: time slew -0.001015 s 2022-03-21T14:48:02.192 INFO:teuthology.orchestra.run.smithi154.stdout:ntpd: time slew -0.001015s 2022-03-21T14:48:02.192 INFO:teuthology.orchestra.run.smithi154.stderr:21 Mar 14:48:02 ntpd[9563]: can't open /var/log/ntpstats/loopstats.20220321: Permission denied 2022-03-21T14:48:02.293 INFO:teuthology.orchestra.run.smithi154.stdout: remote refid st t when poll reach delay offset jitter 2022-03-21T14:48:02.293 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================== 2022-03-21T14:48:02.294 INFO:teuthology.orchestra.run.smithi154.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:02.294 INFO:teuthology.orchestra.run.smithi154.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:02.294 INFO:teuthology.orchestra.run.smithi154.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:02.294 INFO:teuthology.orchestra.run.smithi154.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-21T14:48:02.295 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-21T14:48:02.311 INFO:tasks.kubeadm:Kubeadm start 2022-03-21T14:48:02.311 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-03-21T14:48:02.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-21T14:48:02.315 DEBUG:teuthology.orchestra.run.smithi065:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-21T14:48:02.317 DEBUG:teuthology.orchestra.run.smithi154:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-21T14:48:02.340 INFO:teuthology.orchestra.run.smithi043.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-21T14:48:02.340 INFO:teuthology.orchestra.run.smithi154.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-21T14:48:02.342 INFO:teuthology.orchestra.run.smithi065.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-21T14:48:02.350 INFO:teuthology.orchestra.run.smithi043.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-21T14:48:02.351 INFO:teuthology.orchestra.run.smithi154.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-21T14:48:02.351 INFO:teuthology.orchestra.run.smithi065.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-21T14:48:02.359 INFO:teuthology.orchestra.run.smithi043.stdout:net.ipv4.ip_forward = 1 2022-03-21T14:48:02.360 INFO:teuthology.orchestra.run.smithi065.stdout:net.ipv4.ip_forward = 1 2022-03-21T14:48:02.361 INFO:teuthology.orchestra.run.smithi154.stdout:net.ipv4.ip_forward = 1 2022-03-21T14:48:02.431 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:48:02.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-21T14:48:02.490 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:48:02.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/docker/daemon.json 2022-03-21T14:48:02.503 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:48:02.503 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-21T14:48:02.520 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:48:02.520 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd of=/etc/docker/daemon.json 2022-03-21T14:48:02.580 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:48:02.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-21T14:48:02.596 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:48:02.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/docker/daemon.json 2022-03-21T14:48:02.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl restart docker || true 2022-03-21T14:48:02.658 DEBUG:teuthology.orchestra.run.smithi065:> sudo systemctl restart docker || true 2022-03-21T14:48:02.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl restart docker || true 2022-03-21T14:48:03.647 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-21T14:48:03.654 INFO:tasks.kubeadm:172.21.15.43 is in 172.21.0.0/20 at pos 3882 2022-03-21T14:48:03.667 INFO:tasks.kubeadm:Pod subnet: 10.249.80.0/21 2022-03-21T14:48:03.667 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-21T14:48:03.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-21T14:48:03.671 DEBUG:teuthology.orchestra.run.smithi065:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-21T14:48:03.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-21T14:48:03.700 INFO:teuthology.orchestra.run.smithi065.stderr: 2022-03-21T14:48:03.700 INFO:teuthology.orchestra.run.smithi065.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:03.700 INFO:teuthology.orchestra.run.smithi065.stderr: 2022-03-21T14:48:03.703 INFO:teuthology.orchestra.run.smithi154.stderr: 2022-03-21T14:48:03.703 INFO:teuthology.orchestra.run.smithi154.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:03.703 INFO:teuthology.orchestra.run.smithi154.stderr: 2022-03-21T14:48:03.705 INFO:teuthology.orchestra.run.smithi043.stderr: 2022-03-21T14:48:03.706 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:03.706 INFO:teuthology.orchestra.run.smithi043.stderr: 2022-03-21T14:48:03.822 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:48:03.822 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-21T14:48:03.830 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-21T14:48:03.850 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-21T14:48:03.938 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-21T14:48:03.938 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:48:03.939 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:48:03.939 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-21T14:48:04.006 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-21T14:48:04.007 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-21T14:48:04.087 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-21T14:48:04.087 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-21T14:48:05.334 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2022-03-21T14:48:05.410 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2022-03-21T14:48:05.468 INFO:teuthology.orchestra.run.smithi065.stdout:Reading package lists... 2022-03-21T14:48:05.512 INFO:teuthology.orchestra.run.smithi043.stdout:Building dependency tree... 2022-03-21T14:48:05.513 INFO:teuthology.orchestra.run.smithi043.stdout:Reading state information... 2022-03-21T14:48:05.584 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2022-03-21T14:48:05.584 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2022-03-21T14:48:05.604 INFO:teuthology.orchestra.run.smithi043.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-21T14:48:05.620 INFO:teuthology.orchestra.run.smithi043.stderr: 2022-03-21T14:48:05.620 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:05.621 INFO:teuthology.orchestra.run.smithi043.stderr: 2022-03-21T14:48:05.643 INFO:teuthology.orchestra.run.smithi065.stdout:Building dependency tree... 2022-03-21T14:48:05.644 INFO:teuthology.orchestra.run.smithi065.stdout:Reading state information... 2022-03-21T14:48:05.674 INFO:teuthology.orchestra.run.smithi154.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-21T14:48:05.675 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2022-03-21T14:48:05.687 INFO:teuthology.orchestra.run.smithi154.stderr: 2022-03-21T14:48:05.688 INFO:teuthology.orchestra.run.smithi154.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:05.688 INFO:teuthology.orchestra.run.smithi154.stderr: 2022-03-21T14:48:05.734 INFO:teuthology.orchestra.run.smithi065.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-21T14:48:05.741 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2022-03-21T14:48:05.749 INFO:teuthology.orchestra.run.smithi065.stderr: 2022-03-21T14:48:05.750 INFO:teuthology.orchestra.run.smithi065.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:05.750 INFO:teuthology.orchestra.run.smithi065.stderr: 2022-03-21T14:48:05.803 INFO:teuthology.orchestra.run.smithi065.stdout:Reading package lists... 2022-03-21T14:48:05.855 INFO:teuthology.orchestra.run.smithi043.stdout:Building dependency tree... 2022-03-21T14:48:05.855 INFO:teuthology.orchestra.run.smithi043.stdout:Reading state information... 2022-03-21T14:48:05.903 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2022-03-21T14:48:05.903 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2022-03-21T14:48:05.982 INFO:teuthology.orchestra.run.smithi065.stdout:Building dependency tree... 2022-03-21T14:48:05.983 INFO:teuthology.orchestra.run.smithi065.stdout:Reading state information... 2022-03-21T14:48:06.047 INFO:teuthology.orchestra.run.smithi043.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-21T14:48:06.047 INFO:teuthology.orchestra.run.smithi043.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-21T14:48:06.047 INFO:teuthology.orchestra.run.smithi043.stdout:The following NEW packages will be installed: 2022-03-21T14:48:06.048 INFO:teuthology.orchestra.run.smithi043.stdout: apt-transport-https 2022-03-21T14:48:06.062 INFO:teuthology.orchestra.run.smithi154.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-21T14:48:06.062 INFO:teuthology.orchestra.run.smithi154.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-21T14:48:06.063 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2022-03-21T14:48:06.063 INFO:teuthology.orchestra.run.smithi154.stdout: apt-transport-https 2022-03-21T14:48:06.122 INFO:teuthology.orchestra.run.smithi154.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-21T14:48:06.122 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 4,680 B of archives. 2022-03-21T14:48:06.123 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-21T14:48:06.123 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-21T14:48:06.172 INFO:teuthology.orchestra.run.smithi065.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-21T14:48:06.172 INFO:teuthology.orchestra.run.smithi065.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-21T14:48:06.173 INFO:teuthology.orchestra.run.smithi065.stdout:The following NEW packages will be installed: 2022-03-21T14:48:06.173 INFO:teuthology.orchestra.run.smithi065.stdout: apt-transport-https 2022-03-21T14:48:06.227 INFO:teuthology.orchestra.run.smithi043.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-21T14:48:06.227 INFO:teuthology.orchestra.run.smithi043.stdout:Need to get 4,680 B of archives. 2022-03-21T14:48:06.227 INFO:teuthology.orchestra.run.smithi043.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-21T14:48:06.227 INFO:teuthology.orchestra.run.smithi043.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-21T14:48:06.237 INFO:teuthology.orchestra.run.smithi065.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-21T14:48:06.237 INFO:teuthology.orchestra.run.smithi065.stdout:Need to get 4,680 B of archives. 2022-03-21T14:48:06.237 INFO:teuthology.orchestra.run.smithi065.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-21T14:48:06.237 INFO:teuthology.orchestra.run.smithi065.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-21T14:48:06.358 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: unable to initialize frontend: Dialog 2022-03-21T14:48:06.358 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-21T14:48:06.359 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: falling back to frontend: Readline 2022-03-21T14:48:06.381 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: unable to initialize frontend: Readline 2022-03-21T14:48:06.381 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-21T14:48:06.381 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: falling back to frontend: Teletype 2022-03-21T14:48:06.387 INFO:teuthology.orchestra.run.smithi154.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-21T14:48:06.434 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 4,680 B in 0s (89.0 kB/s) 2022-03-21T14:48:06.471 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: unable to initialize frontend: Dialog 2022-03-21T14:48:06.472 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-21T14:48:06.473 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: falling back to frontend: Readline 2022-03-21T14:48:06.487 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: unable to initialize frontend: Dialog 2022-03-21T14:48:06.487 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-21T14:48:06.488 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: falling back to frontend: Readline 2022-03-21T14:48:06.491 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: unable to initialize frontend: Readline 2022-03-21T14:48:06.491 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-21T14:48:06.491 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: falling back to frontend: Teletype 2022-03-21T14:48:06.497 INFO:teuthology.orchestra.run.smithi065.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-21T14:48:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: unable to initialize frontend: Readline 2022-03-21T14:48:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-21T14:48:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: falling back to frontend: Teletype 2022-03-21T14:48:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-21T14:48:06.528 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package apt-transport-https. 2022-03-21T14:48:06.550 INFO:teuthology.orchestra.run.smithi065.stdout:Fetched 4,680 B in 0s (85.0 kB/s) 2022-03-21T14:48:06.560 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 4,680 B in 0s (27.6 kB/s) 2022-03-21T14:48:06.570 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-21T14:48:06.574 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-21T14:48:06.618 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-21T14:48:06.650 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package apt-transport-https. 2022-03-21T14:48:06.694 INFO:teuthology.orchestra.run.smithi043.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-21T14:48:06.698 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-21T14:48:06.732 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-21T14:48:06.739 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package apt-transport-https. 2022-03-21T14:48:06.783 INFO:teuthology.orchestra.run.smithi065.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-21T14:48:06.786 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-21T14:48:06.829 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-21T14:48:06.864 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-21T14:48:07.003 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-21T14:48:07.095 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-21T14:48:07.358 INFO:teuthology.orchestra.run.smithi154.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-21T14:48:07.372 INFO:teuthology.orchestra.run.smithi154.stderr: 2022-03-21T14:48:07.372 INFO:teuthology.orchestra.run.smithi154.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:07.373 INFO:teuthology.orchestra.run.smithi154.stderr: 2022-03-21T14:48:07.547 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:48:07.547 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-21T14:48:07.576 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-21T14:48:07.597 INFO:teuthology.orchestra.run.smithi043.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-21T14:48:07.614 INFO:teuthology.orchestra.run.smithi043.stderr: 2022-03-21T14:48:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:07.614 INFO:teuthology.orchestra.run.smithi043.stderr: 2022-03-21T14:48:07.636 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-21T14:48:07.668 INFO:teuthology.orchestra.run.smithi065.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-21T14:48:07.683 INFO:teuthology.orchestra.run.smithi065.stderr: 2022-03-21T14:48:07.684 INFO:teuthology.orchestra.run.smithi065.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:07.684 INFO:teuthology.orchestra.run.smithi065.stderr: 2022-03-21T14:48:07.869 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-21T14:48:07.870 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:48:07.870 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-21T14:48:07.877 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-21T14:48:07.937 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-21T14:48:07.938 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-21T14:48:07.991 INFO:teuthology.orchestra.run.smithi065.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-21T14:48:08.018 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-21T14:48:08.091 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-21T14:48:08.224 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-21T14:48:08.327 INFO:teuthology.orchestra.run.smithi065.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-21T14:48:08.369 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 64.1 kB in 1s (74.7 kB/s) 2022-03-21T14:48:08.443 INFO:teuthology.orchestra.run.smithi043.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-21T14:48:08.519 INFO:teuthology.orchestra.run.smithi065.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-21T14:48:08.549 INFO:teuthology.orchestra.run.smithi043.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-21T14:48:08.661 INFO:teuthology.orchestra.run.smithi065.stdout:Fetched 64.1 kB in 1s (77.5 kB/s) 2022-03-21T14:48:08.697 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 64.1 kB in 1s (68.9 kB/s) 2022-03-21T14:48:09.226 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2022-03-21T14:48:09.401 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2022-03-21T14:48:09.402 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2022-03-21T14:48:09.494 INFO:teuthology.orchestra.run.smithi154.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-21T14:48:09.507 INFO:teuthology.orchestra.run.smithi154.stderr: 2022-03-21T14:48:09.508 INFO:teuthology.orchestra.run.smithi154.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:09.508 INFO:teuthology.orchestra.run.smithi154.stderr: 2022-03-21T14:48:09.545 INFO:teuthology.orchestra.run.smithi065.stdout:Reading package lists... 2022-03-21T14:48:09.566 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2022-03-21T14:48:09.580 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2022-03-21T14:48:09.727 INFO:teuthology.orchestra.run.smithi065.stdout:Building dependency tree... 2022-03-21T14:48:09.727 INFO:teuthology.orchestra.run.smithi065.stdout:Reading state information... 2022-03-21T14:48:09.741 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2022-03-21T14:48:09.741 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2022-03-21T14:48:09.760 INFO:teuthology.orchestra.run.smithi043.stdout:Building dependency tree... 2022-03-21T14:48:09.760 INFO:teuthology.orchestra.run.smithi043.stdout:Reading state information... 2022-03-21T14:48:09.819 INFO:teuthology.orchestra.run.smithi065.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-21T14:48:09.832 INFO:teuthology.orchestra.run.smithi065.stderr: 2022-03-21T14:48:09.832 INFO:teuthology.orchestra.run.smithi065.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:09.832 INFO:teuthology.orchestra.run.smithi065.stderr: 2022-03-21T14:48:09.855 INFO:teuthology.orchestra.run.smithi043.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-21T14:48:09.868 INFO:teuthology.orchestra.run.smithi043.stderr: 2022-03-21T14:48:09.869 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-21T14:48:09.869 INFO:teuthology.orchestra.run.smithi043.stderr: 2022-03-21T14:48:09.881 INFO:teuthology.orchestra.run.smithi154.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-21T14:48:09.881 INFO:teuthology.orchestra.run.smithi154.stdout:bridge-utils set to manually installed. 2022-03-21T14:48:09.881 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2022-03-21T14:48:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-21T14:48:09.882 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2022-03-21T14:48:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: nftables 2022-03-21T14:48:09.887 INFO:teuthology.orchestra.run.smithi065.stdout:Reading package lists... 2022-03-21T14:48:09.923 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2022-03-21T14:48:09.934 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2022-03-21T14:48:09.935 INFO:teuthology.orchestra.run.smithi154.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-21T14:48:10.003 INFO:teuthology.orchestra.run.smithi154.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-21T14:48:10.003 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 77.7 MB of archives. 2022-03-21T14:48:10.003 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-21T14:48:10.004 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-21T14:48:10.043 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-21T14:48:10.052 INFO:teuthology.orchestra.run.smithi065.stdout:Building dependency tree... 2022-03-21T14:48:10.053 INFO:teuthology.orchestra.run.smithi065.stdout:Reading state information... 2022-03-21T14:48:10.069 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-21T14:48:10.102 INFO:teuthology.orchestra.run.smithi043.stdout:Building dependency tree... 2022-03-21T14:48:10.103 INFO:teuthology.orchestra.run.smithi043.stdout:Reading state information... 2022-03-21T14:48:10.193 INFO:teuthology.orchestra.run.smithi065.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-21T14:48:10.193 INFO:teuthology.orchestra.run.smithi065.stdout:bridge-utils set to manually installed. 2022-03-21T14:48:10.193 INFO:teuthology.orchestra.run.smithi065.stdout:The following additional packages will be installed: 2022-03-21T14:48:10.194 INFO:teuthology.orchestra.run.smithi065.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-21T14:48:10.194 INFO:teuthology.orchestra.run.smithi065.stdout:Suggested packages: 2022-03-21T14:48:10.195 INFO:teuthology.orchestra.run.smithi065.stdout: nftables 2022-03-21T14:48:10.216 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-21T14:48:10.243 INFO:teuthology.orchestra.run.smithi065.stdout:The following NEW packages will be installed: 2022-03-21T14:48:10.244 INFO:teuthology.orchestra.run.smithi065.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-21T14:48:10.248 INFO:teuthology.orchestra.run.smithi043.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-21T14:48:10.248 INFO:teuthology.orchestra.run.smithi043.stdout:bridge-utils set to manually installed. 2022-03-21T14:48:10.249 INFO:teuthology.orchestra.run.smithi043.stdout:The following additional packages will be installed: 2022-03-21T14:48:10.249 INFO:teuthology.orchestra.run.smithi043.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-21T14:48:10.250 INFO:teuthology.orchestra.run.smithi043.stdout:Suggested packages: 2022-03-21T14:48:10.250 INFO:teuthology.orchestra.run.smithi043.stdout: nftables 2022-03-21T14:48:10.301 INFO:teuthology.orchestra.run.smithi043.stdout:The following NEW packages will be installed: 2022-03-21T14:48:10.302 INFO:teuthology.orchestra.run.smithi043.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-21T14:48:10.330 INFO:teuthology.orchestra.run.smithi065.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-21T14:48:10.330 INFO:teuthology.orchestra.run.smithi065.stdout:Need to get 77.7 MB of archives. 2022-03-21T14:48:10.330 INFO:teuthology.orchestra.run.smithi065.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-21T14:48:10.331 INFO:teuthology.orchestra.run.smithi065.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-21T14:48:10.386 INFO:teuthology.orchestra.run.smithi065.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-21T14:48:10.420 INFO:teuthology.orchestra.run.smithi065.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-21T14:48:10.504 INFO:teuthology.orchestra.run.smithi043.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-21T14:48:10.504 INFO:teuthology.orchestra.run.smithi043.stdout:Need to get 77.7 MB of archives. 2022-03-21T14:48:10.504 INFO:teuthology.orchestra.run.smithi043.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-21T14:48:10.505 INFO:teuthology.orchestra.run.smithi043.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-21T14:48:10.537 INFO:teuthology.orchestra.run.smithi065.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-21T14:48:10.613 INFO:teuthology.orchestra.run.smithi043.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-21T14:48:10.668 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-21T14:48:10.673 INFO:teuthology.orchestra.run.smithi043.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-21T14:48:10.788 INFO:teuthology.orchestra.run.smithi043.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-21T14:48:11.511 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-21T14:48:11.565 INFO:teuthology.orchestra.run.smithi043.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-21T14:48:11.761 INFO:teuthology.orchestra.run.smithi065.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-21T14:48:12.158 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-21T14:48:12.158 INFO:teuthology.orchestra.run.smithi043.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-21T14:48:12.383 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-21T14:48:12.585 INFO:teuthology.orchestra.run.smithi043.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-21T14:48:12.717 INFO:teuthology.orchestra.run.smithi043.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-21T14:48:12.868 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: unable to initialize frontend: Dialog 2022-03-21T14:48:12.868 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-21T14:48:12.869 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: falling back to frontend: Readline 2022-03-21T14:48:12.875 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: unable to initialize frontend: Readline 2022-03-21T14:48:12.876 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-21T14:48:12.876 INFO:teuthology.orchestra.run.smithi154.stderr:debconf: falling back to frontend: Teletype 2022-03-21T14:48:12.881 INFO:teuthology.orchestra.run.smithi154.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-21T14:48:12.931 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 77.7 MB in 3s (28.7 MB/s) 2022-03-21T14:48:13.019 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package conntrack. 2022-03-21T14:48:13.064 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-21T14:48:13.068 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-21T14:48:13.112 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: unable to initialize frontend: Dialog 2022-03-21T14:48:13.113 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-21T14:48:13.113 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: falling back to frontend: Readline 2022-03-21T14:48:13.117 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-21T14:48:13.121 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: unable to initialize frontend: Readline 2022-03-21T14:48:13.121 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-21T14:48:13.121 INFO:teuthology.orchestra.run.smithi043.stderr:debconf: falling back to frontend: Teletype 2022-03-21T14:48:13.126 INFO:teuthology.orchestra.run.smithi043.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-21T14:48:13.174 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 77.7 MB in 3s (30.2 MB/s) 2022-03-21T14:48:13.265 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package conntrack. 2022-03-21T14:48:13.308 INFO:teuthology.orchestra.run.smithi043.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-21T14:48:13.311 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-21T14:48:13.342 INFO:teuthology.orchestra.run.smithi065.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-21T14:48:13.355 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-21T14:48:13.412 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package cri-tools. 2022-03-21T14:48:13.427 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-21T14:48:13.471 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-21T14:48:13.675 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package cri-tools. 2022-03-21T14:48:13.690 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-21T14:48:13.733 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-21T14:48:13.904 INFO:teuthology.orchestra.run.smithi065.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-21T14:48:14.070 INFO:teuthology.orchestra.run.smithi065.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-21T14:48:14.480 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: unable to initialize frontend: Dialog 2022-03-21T14:48:14.480 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-21T14:48:14.481 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: falling back to frontend: Readline 2022-03-21T14:48:14.488 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: unable to initialize frontend: Readline 2022-03-21T14:48:14.489 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-21T14:48:14.489 INFO:teuthology.orchestra.run.smithi065.stderr:debconf: falling back to frontend: Teletype 2022-03-21T14:48:14.494 INFO:teuthology.orchestra.run.smithi065.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-21T14:48:14.546 INFO:teuthology.orchestra.run.smithi065.stdout:Fetched 77.7 MB in 4s (19.4 MB/s) 2022-03-21T14:48:14.620 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package conntrack. 2022-03-21T14:48:14.668 INFO:teuthology.orchestra.run.smithi065.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-21T14:48:14.671 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-21T14:48:14.710 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-21T14:48:14.768 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ebtables. 2022-03-21T14:48:14.783 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-21T14:48:14.835 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-21T14:48:15.080 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-21T14:48:15.081 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ebtables. 2022-03-21T14:48:15.091 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package cri-tools. 2022-03-21T14:48:15.092 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-21T14:48:15.093 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-21T14:48:15.105 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-21T14:48:15.121 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-21T14:48:15.141 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-21T14:48:15.147 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-21T14:48:15.434 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-21T14:48:15.447 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-21T14:48:15.492 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-21T14:48:16.461 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package ebtables. 2022-03-21T14:48:16.476 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-21T14:48:16.552 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-21T14:48:16.574 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package socat. 2022-03-21T14:48:16.587 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-21T14:48:16.632 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-21T14:48:16.856 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-21T14:48:16.870 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-21T14:48:16.914 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-21T14:48:17.004 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package socat. 2022-03-21T14:48:17.016 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-21T14:48:17.036 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package kubelet. 2022-03-21T14:48:17.050 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-21T14:48:17.053 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-21T14:48:17.094 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-21T14:48:17.482 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package kubelet. 2022-03-21T14:48:17.495 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-21T14:48:17.532 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-21T14:48:18.400 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package socat. 2022-03-21T14:48:18.413 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-21T14:48:18.450 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-21T14:48:18.839 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package kubelet. 2022-03-21T14:48:18.854 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-21T14:48:18.897 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-21T14:48:19.532 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package kubectl. 2022-03-21T14:48:19.546 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-21T14:48:19.591 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-21T14:48:20.028 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package kubectl. 2022-03-21T14:48:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-21T14:48:20.078 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-21T14:48:20.760 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package kubeadm. 2022-03-21T14:48:20.774 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-21T14:48:20.818 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-21T14:48:21.289 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package kubeadm. 2022-03-21T14:48:21.298 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-21T14:48:21.300 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package kubectl. 2022-03-21T14:48:21.314 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-21T14:48:21.339 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-21T14:48:21.359 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-21T14:48:21.976 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-21T14:48:22.089 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-21T14:48:22.206 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-21T14:48:22.488 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-21T14:48:22.586 INFO:teuthology.orchestra.run.smithi065.stdout:Selecting previously unselected package kubeadm. 2022-03-21T14:48:22.601 INFO:teuthology.orchestra.run.smithi065.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-21T14:48:22.627 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-21T14:48:22.644 INFO:teuthology.orchestra.run.smithi065.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-21T14:48:22.753 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-21T14:48:23.004 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-21T14:48:23.114 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-21T14:48:23.250 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-21T14:48:23.378 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-21T14:48:23.543 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-21T14:48:23.595 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-21T14:48:23.695 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-21T14:48:23.791 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-21T14:48:24.615 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-21T14:48:24.616 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-21T14:48:24.616 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-21T14:48:24.616 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-21T14:48:24.617 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-21T14:48:24.617 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-21T14:48:24.617 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-21T14:48:24.712 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-21T14:48:24.827 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-21T14:48:24.947 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-21T14:48:25.090 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-21T14:48:25.101 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-21T14:48:25.217 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-21T14:48:25.449 INFO:teuthology.orchestra.run.smithi065.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-21T14:48:25.948 INFO:teuthology.orchestra.run.smithi065.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-21T14:48:26.321 INFO:teuthology.orchestra.run.smithi065.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-21T14:48:27.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-21T14:48:27.211 DEBUG:teuthology.orchestra.run.smithi065:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-21T14:48:27.249 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-21T14:48:39.762 INFO:teuthology.orchestra.run.smithi154.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-21T14:48:40.547 INFO:teuthology.orchestra.run.smithi043.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-21T14:48:41.447 INFO:teuthology.orchestra.run.smithi065.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-21T14:48:42.840 INFO:teuthology.orchestra.run.smithi154.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-21T14:48:43.916 INFO:teuthology.orchestra.run.smithi043.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-21T14:48:44.858 INFO:teuthology.orchestra.run.smithi065.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-21T14:48:45.352 INFO:teuthology.orchestra.run.smithi154.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-21T14:48:46.946 INFO:teuthology.orchestra.run.smithi043.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-21T14:48:47.472 INFO:teuthology.orchestra.run.smithi065.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-21T14:48:49.677 INFO:teuthology.orchestra.run.smithi154.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-21T14:48:50.935 INFO:teuthology.orchestra.run.smithi154.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-21T14:48:51.445 INFO:teuthology.orchestra.run.smithi043.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-21T14:48:51.613 INFO:teuthology.orchestra.run.smithi065.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-21T14:48:53.153 INFO:teuthology.orchestra.run.smithi043.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-21T14:48:53.229 INFO:teuthology.orchestra.run.smithi065.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-21T14:48:56.451 INFO:teuthology.orchestra.run.smithi154.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-21T14:48:58.800 INFO:teuthology.orchestra.run.smithi154.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-21T14:49:00.028 INFO:teuthology.orchestra.run.smithi065.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-21T14:49:00.702 INFO:teuthology.orchestra.run.smithi043.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-21T14:49:02.428 INFO:teuthology.orchestra.run.smithi065.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-21T14:49:03.293 INFO:teuthology.orchestra.run.smithi043.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-21T14:49:03.298 INFO:tasks.kubeadm:Token: abcdef.4903bv6e2hu9govx 2022-03-21T14:49:03.298 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): '172.21.15.43', Remote(name='ubuntu@smithi065.front.sepia.ceph.com'): '172.21.15.65', Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): '172.21.15.154'} 2022-03-21T14:49:03.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo kubeadm init --node-name smithi043 --token abcdef.4903bv6e2hu9govx --pod-network-cidr 10.249.80.0/21 2022-03-21T14:49:03.497 INFO:teuthology.orchestra.run.smithi043.stdout:[init] Using Kubernetes version: v1.23.5 2022-03-21T14:49:03.498 INFO:teuthology.orchestra.run.smithi043.stdout:[preflight] Running pre-flight checks 2022-03-21T14:49:03.920 INFO:teuthology.orchestra.run.smithi043.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-21T14:49:03.920 INFO:teuthology.orchestra.run.smithi043.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-21T14:49:03.920 INFO:teuthology.orchestra.run.smithi043.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-21T14:49:04.241 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-21T14:49:04.333 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "ca" certificate and key 2022-03-21T14:49:04.571 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "apiserver" certificate and key 2022-03-21T14:49:04.572 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi043] and IPs [10.96.0.1 172.21.15.43] 2022-03-21T14:49:04.649 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-21T14:49:04.844 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-21T14:49:05.279 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-21T14:49:05.461 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-21T14:49:05.559 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-21T14:49:05.560 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi043] and IPs [172.21.15.43 127.0.0.1 ::1] 2022-03-21T14:49:05.723 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-21T14:49:05.723 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi043] and IPs [172.21.15.43 127.0.0.1 ::1] 2022-03-21T14:49:05.960 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-21T14:49:06.408 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-21T14:49:06.772 INFO:teuthology.orchestra.run.smithi043.stdout:[certs] Generating "sa" key and public key 2022-03-21T14:49:06.773 INFO:teuthology.orchestra.run.smithi043.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-21T14:49:07.144 INFO:teuthology.orchestra.run.smithi043.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-21T14:49:07.376 INFO:teuthology.orchestra.run.smithi043.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-21T14:49:07.649 INFO:teuthology.orchestra.run.smithi043.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-21T14:49:07.837 INFO:teuthology.orchestra.run.smithi043.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-21T14:49:07.850 INFO:teuthology.orchestra.run.smithi043.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-21T14:49:07.852 INFO:teuthology.orchestra.run.smithi043.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-21T14:49:07.852 INFO:teuthology.orchestra.run.smithi043.stdout:[kubelet-start] Starting the kubelet 2022-03-21T14:49:08.067 INFO:teuthology.orchestra.run.smithi043.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-21T14:49:08.067 INFO:teuthology.orchestra.run.smithi043.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-21T14:49:08.070 INFO:teuthology.orchestra.run.smithi043.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-21T14:49:08.071 INFO:teuthology.orchestra.run.smithi043.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-21T14:49:08.072 INFO:teuthology.orchestra.run.smithi043.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-21T14:49:08.074 INFO:teuthology.orchestra.run.smithi043.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-21T14:49:24.597 INFO:teuthology.orchestra.run.smithi043.stdout:[apiclient] All control plane components are healthy after 16.522525 seconds 2022-03-21T14:49:24.597 INFO:teuthology.orchestra.run.smithi043.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-21T14:49:24.620 INFO:teuthology.orchestra.run.smithi043.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-21T14:49:24.620 INFO:teuthology.orchestra.run.smithi043.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-21T14:49:25.187 INFO:teuthology.orchestra.run.smithi043.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-21T14:49:25.187 INFO:teuthology.orchestra.run.smithi043.stdout:[mark-control-plane] Marking the node smithi043 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-21T14:49:25.188 INFO:teuthology.orchestra.run.smithi043.stdout:[mark-control-plane] Marking the node smithi043 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-21T14:49:25.710 INFO:teuthology.orchestra.run.smithi043.stdout:[bootstrap-token] Using token: abcdef.4903bv6e2hu9govx 2022-03-21T14:49:25.710 INFO:teuthology.orchestra.run.smithi043.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-21T14:49:25.742 INFO:teuthology.orchestra.run.smithi043.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-21T14:49:25.770 INFO:teuthology.orchestra.run.smithi043.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-21T14:49:25.778 INFO:teuthology.orchestra.run.smithi043.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-21T14:49:25.787 INFO:teuthology.orchestra.run.smithi043.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-21T14:49:25.795 INFO:teuthology.orchestra.run.smithi043.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-21T14:49:25.917 INFO:teuthology.orchestra.run.smithi043.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-21T14:49:26.390 INFO:teuthology.orchestra.run.smithi043.stdout:[addons] Applied essential addon: CoreDNS 2022-03-21T14:49:26.633 INFO:teuthology.orchestra.run.smithi043.stdout:[addons] Applied essential addon: kube-proxy 2022-03-21T14:49:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:49:26.636 INFO:teuthology.orchestra.run.smithi043.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-21T14:49:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:49:26.636 INFO:teuthology.orchestra.run.smithi043.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-21T14:49:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:49:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: mkdir -p $HOME/.kube 2022-03-21T14:49:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-21T14:49:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-21T14:49:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:49:26.638 INFO:teuthology.orchestra.run.smithi043.stdout:Alternatively, if you are the root user, you can run: 2022-03-21T14:49:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:49:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-21T14:49:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:49:26.639 INFO:teuthology.orchestra.run.smithi043.stdout:You should now deploy a pod network to the cluster. 2022-03-21T14:49:26.639 INFO:teuthology.orchestra.run.smithi043.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-21T14:49:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-21T14:49:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:49:26.640 INFO:teuthology.orchestra.run.smithi043.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-21T14:49:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:49:26.640 INFO:teuthology.orchestra.run.smithi043.stdout:kubeadm join 172.21.15.43:6443 --token abcdef.4903bv6e2hu9govx \ 2022-03-21T14:49:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: --discovery-token-ca-cert-hash sha256:e3dd092ad6bd9611485fb3be98d70edd4c5bc2fa50e2cf5d457c570254bfa742 2022-03-21T14:49:26.642 DEBUG:teuthology.orchestra.run.smithi065:> sudo kubeadm join 172.21.15.43:6443 --node-name smithi065 --token abcdef.4903bv6e2hu9govx --discovery-token-unsafe-skip-ca-verification 2022-03-21T14:49:26.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo kubeadm join 172.21.15.43:6443 --node-name smithi154 --token abcdef.4903bv6e2hu9govx --discovery-token-unsafe-skip-ca-verification 2022-03-21T14:49:26.686 INFO:teuthology.orchestra.run.smithi065.stdout:[preflight] Running pre-flight checks 2022-03-21T14:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout:[preflight] Running pre-flight checks 2022-03-21T14:49:39.480 INFO:teuthology.orchestra.run.smithi154.stdout:[preflight] Reading configuration from the cluster... 2022-03-21T14:49:39.480 INFO:teuthology.orchestra.run.smithi154.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-21T14:49:39.481 INFO:teuthology.orchestra.run.smithi065.stdout:[preflight] Reading configuration from the cluster... 2022-03-21T14:49:39.481 INFO:teuthology.orchestra.run.smithi065.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-21T14:49:39.842 INFO:teuthology.orchestra.run.smithi154.stderr:W0321 14:49:39.844167 11173 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-21T14:49:39.842 INFO:teuthology.orchestra.run.smithi065.stderr:W0321 14:49:39.844201 11147 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-21T14:49:39.864 INFO:teuthology.orchestra.run.smithi065.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-21T14:49:39.864 INFO:teuthology.orchestra.run.smithi065.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-21T14:49:39.864 INFO:teuthology.orchestra.run.smithi065.stdout:[kubelet-start] Starting the kubelet 2022-03-21T14:49:39.865 INFO:teuthology.orchestra.run.smithi154.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-21T14:49:39.958 INFO:teuthology.orchestra.run.smithi154.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-21T14:49:39.958 INFO:teuthology.orchestra.run.smithi154.stdout:[kubelet-start] Starting the kubelet 2022-03-21T14:49:40.100 INFO:teuthology.orchestra.run.smithi065.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-21T14:49:40.173 INFO:teuthology.orchestra.run.smithi154.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-21T14:49:45.640 INFO:teuthology.orchestra.run.smithi065.stdout: 2022-03-21T14:49:45.641 INFO:teuthology.orchestra.run.smithi065.stdout:This node has joined the cluster: 2022-03-21T14:49:45.641 INFO:teuthology.orchestra.run.smithi065.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-21T14:49:45.641 INFO:teuthology.orchestra.run.smithi065.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-21T14:49:45.641 INFO:teuthology.orchestra.run.smithi065.stdout: 2022-03-21T14:49:45.642 INFO:teuthology.orchestra.run.smithi065.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-21T14:49:45.642 INFO:teuthology.orchestra.run.smithi065.stdout: 2022-03-21T14:49:54.193 INFO:teuthology.orchestra.run.smithi154.stdout: 2022-03-21T14:49:54.193 INFO:teuthology.orchestra.run.smithi154.stdout:This node has joined the cluster: 2022-03-21T14:49:54.193 INFO:teuthology.orchestra.run.smithi154.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-21T14:49:54.194 INFO:teuthology.orchestra.run.smithi154.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-21T14:49:54.194 INFO:teuthology.orchestra.run.smithi154.stdout: 2022-03-21T14:49:54.194 INFO:teuthology.orchestra.run.smithi154.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-21T14:49:54.194 INFO:teuthology.orchestra.run.smithi154.stdout: 2022-03-21T14:49:54.198 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:49:54.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-21T14:49:54.214 INFO:tasks.kubeadm:Setting up kubectl 2022-03-21T14:49:54.215 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-21T14:49:54.230 DEBUG:teuthology.orchestra.run.smithi065:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-21T14:49:54.248 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-21T14:49:54.265 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:49:54.265 DEBUG:teuthology.orchestra.run.smithi043:> dd of=.kube/config 2022-03-21T14:49:54.272 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:49:54.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/root/.kube/config 2022-03-21T14:49:54.332 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:49:54.332 DEBUG:teuthology.orchestra.run.smithi065:> dd of=.kube/config 2022-03-21T14:49:54.340 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:49:54.340 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd of=/root/.kube/config 2022-03-21T14:49:54.397 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:49:54.398 DEBUG:teuthology.orchestra.run.smithi154:> dd of=.kube/config 2022-03-21T14:49:54.406 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:49:54.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/root/.kube/config 2022-03-21T14:49:54.418 DEBUG:teuthology.orchestra.run.smithi043:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-03-21T14:49:54.435 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-03-21T14:49:54.435 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2022-03-21T14:49:54.559 INFO:teuthology.orchestra.run.smithi043.stdout:--- 2022-03-21T14:49:54.559 INFO:teuthology.orchestra.run.smithi043.stdout:apiVersion: policy/v1beta1 2022-03-21T14:49:54.560 INFO:teuthology.orchestra.run.smithi043.stdout:kind: PodSecurityPolicy 2022-03-21T14:49:54.560 INFO:teuthology.orchestra.run.smithi043.stdout:metadata: 2022-03-21T14:49:54.560 INFO:teuthology.orchestra.run.smithi043.stdout: name: psp.flannel.unprivileged 2022-03-21T14:49:54.560 INFO:teuthology.orchestra.run.smithi043.stdout: annotations: 2022-03-21T14:49:54.560 INFO:teuthology.orchestra.run.smithi043.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-03-21T14:49:54.561 INFO:teuthology.orchestra.run.smithi043.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-03-21T14:49:54.561 INFO:teuthology.orchestra.run.smithi043.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-03-21T14:49:54.561 INFO:teuthology.orchestra.run.smithi043.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-03-21T14:49:54.561 INFO:teuthology.orchestra.run.smithi043.stdout:spec: 2022-03-21T14:49:54.562 INFO:teuthology.orchestra.run.smithi043.stdout: privileged: false 2022-03-21T14:49:54.562 INFO:teuthology.orchestra.run.smithi043.stdout: volumes: 2022-03-21T14:49:54.562 INFO:teuthology.orchestra.run.smithi043.stdout: - configMap 2022-03-21T14:49:54.562 INFO:teuthology.orchestra.run.smithi043.stdout: - secret 2022-03-21T14:49:54.562 INFO:teuthology.orchestra.run.smithi043.stdout: - emptyDir 2022-03-21T14:49:54.563 INFO:teuthology.orchestra.run.smithi043.stdout: - hostPath 2022-03-21T14:49:54.563 INFO:teuthology.orchestra.run.smithi043.stdout: allowedHostPaths: 2022-03-21T14:49:54.563 INFO:teuthology.orchestra.run.smithi043.stdout: - pathPrefix: "/etc/cni/net.d" 2022-03-21T14:49:54.563 INFO:teuthology.orchestra.run.smithi043.stdout: - pathPrefix: "/etc/kube-flannel" 2022-03-21T14:49:54.563 INFO:teuthology.orchestra.run.smithi043.stdout: - pathPrefix: "/run/flannel" 2022-03-21T14:49:54.564 INFO:teuthology.orchestra.run.smithi043.stdout: readOnlyRootFilesystem: false 2022-03-21T14:49:54.564 INFO:teuthology.orchestra.run.smithi043.stdout: # Users and groups 2022-03-21T14:49:54.564 INFO:teuthology.orchestra.run.smithi043.stdout: runAsUser: 2022-03-21T14:49:54.564 INFO:teuthology.orchestra.run.smithi043.stdout: rule: RunAsAny 2022-03-21T14:49:54.565 INFO:teuthology.orchestra.run.smithi043.stdout: supplementalGroups: 2022-03-21T14:49:54.565 INFO:teuthology.orchestra.run.smithi043.stdout: rule: RunAsAny 2022-03-21T14:49:54.565 INFO:teuthology.orchestra.run.smithi043.stdout: fsGroup: 2022-03-21T14:49:54.565 INFO:teuthology.orchestra.run.smithi043.stdout: rule: RunAsAny 2022-03-21T14:49:54.566 INFO:teuthology.orchestra.run.smithi043.stdout: # Privilege Escalation 2022-03-21T14:49:54.566 INFO:teuthology.orchestra.run.smithi043.stdout: allowPrivilegeEscalation: false 2022-03-21T14:49:54.566 INFO:teuthology.orchestra.run.smithi043.stdout: defaultAllowPrivilegeEscalation: false 2022-03-21T14:49:54.566 INFO:teuthology.orchestra.run.smithi043.stdout: # Capabilities 2022-03-21T14:49:54.566 INFO:teuthology.orchestra.run.smithi043.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-03-21T14:49:54.567 INFO:teuthology.orchestra.run.smithi043.stdout: defaultAddCapabilities: [] 2022-03-21T14:49:54.567 INFO:teuthology.orchestra.run.smithi043.stdout: requiredDropCapabilities: [] 2022-03-21T14:49:54.567 INFO:teuthology.orchestra.run.smithi043.stdout: # Host namespaces 2022-03-21T14:49:54.567 INFO:teuthology.orchestra.run.smithi043.stdout: hostPID: false 2022-03-21T14:49:54.567 INFO:teuthology.orchestra.run.smithi043.stdout: hostIPC: false 2022-03-21T14:49:54.568 INFO:teuthology.orchestra.run.smithi043.stdout: hostNetwork: true 2022-03-21T14:49:54.568 INFO:teuthology.orchestra.run.smithi043.stdout: hostPorts: 2022-03-21T14:49:54.568 INFO:teuthology.orchestra.run.smithi043.stdout: - min: 0 2022-03-21T14:49:54.568 INFO:teuthology.orchestra.run.smithi043.stdout: max: 65535 2022-03-21T14:49:54.569 INFO:teuthology.orchestra.run.smithi043.stdout: # SELinux 2022-03-21T14:49:54.569 INFO:teuthology.orchestra.run.smithi043.stdout: seLinux: 2022-03-21T14:49:54.569 INFO:teuthology.orchestra.run.smithi043.stdout: # SELinux is unused in CaaSP 2022-03-21T14:49:54.569 INFO:teuthology.orchestra.run.smithi043.stdout: rule: 'RunAsAny' 2022-03-21T14:49:54.569 INFO:teuthology.orchestra.run.smithi043.stdout:--- 2022-03-21T14:49:54.570 INFO:teuthology.orchestra.run.smithi043.stdout:kind: ClusterRole 2022-03-21T14:49:54.570 INFO:teuthology.orchestra.run.smithi043.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-21T14:49:54.570 INFO:teuthology.orchestra.run.smithi043.stdout:metadata: 2022-03-21T14:49:54.570 INFO:teuthology.orchestra.run.smithi043.stdout: name: flannel 2022-03-21T14:49:54.570 INFO:teuthology.orchestra.run.smithi043.stdout:rules: 2022-03-21T14:49:54.571 INFO:teuthology.orchestra.run.smithi043.stdout:- apiGroups: ['extensions'] 2022-03-21T14:49:54.571 INFO:teuthology.orchestra.run.smithi043.stdout: resources: ['podsecuritypolicies'] 2022-03-21T14:49:54.571 INFO:teuthology.orchestra.run.smithi043.stdout: verbs: ['use'] 2022-03-21T14:49:54.571 INFO:teuthology.orchestra.run.smithi043.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-03-21T14:49:54.572 INFO:teuthology.orchestra.run.smithi043.stdout:- apiGroups: 2022-03-21T14:49:54.572 INFO:teuthology.orchestra.run.smithi043.stdout: - "" 2022-03-21T14:49:54.572 INFO:teuthology.orchestra.run.smithi043.stdout: resources: 2022-03-21T14:49:54.572 INFO:teuthology.orchestra.run.smithi043.stdout: - pods 2022-03-21T14:49:54.572 INFO:teuthology.orchestra.run.smithi043.stdout: verbs: 2022-03-21T14:49:54.573 INFO:teuthology.orchestra.run.smithi043.stdout: - get 2022-03-21T14:49:54.573 INFO:teuthology.orchestra.run.smithi043.stdout:- apiGroups: 2022-03-21T14:49:54.573 INFO:teuthology.orchestra.run.smithi043.stdout: - "" 2022-03-21T14:49:54.573 INFO:teuthology.orchestra.run.smithi043.stdout: resources: 2022-03-21T14:49:54.573 INFO:teuthology.orchestra.run.smithi043.stdout: - nodes 2022-03-21T14:49:54.574 INFO:teuthology.orchestra.run.smithi043.stdout: verbs: 2022-03-21T14:49:54.574 INFO:teuthology.orchestra.run.smithi043.stdout: - list 2022-03-21T14:49:54.574 INFO:teuthology.orchestra.run.smithi043.stdout: - watch 2022-03-21T14:49:54.574 INFO:teuthology.orchestra.run.smithi043.stdout:- apiGroups: 2022-03-21T14:49:54.575 INFO:teuthology.orchestra.run.smithi043.stdout: - "" 2022-03-21T14:49:54.575 INFO:teuthology.orchestra.run.smithi043.stdout: resources: 2022-03-21T14:49:54.575 INFO:teuthology.orchestra.run.smithi043.stdout: - nodes/status 2022-03-21T14:49:54.575 INFO:teuthology.orchestra.run.smithi043.stdout: verbs: 2022-03-21T14:49:54.575 INFO:teuthology.orchestra.run.smithi043.stdout: - patch 2022-03-21T14:49:54.576 INFO:teuthology.orchestra.run.smithi043.stdout:--- 2022-03-21T14:49:54.576 INFO:teuthology.orchestra.run.smithi043.stdout:kind: ClusterRoleBinding 2022-03-21T14:49:54.576 INFO:teuthology.orchestra.run.smithi043.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-21T14:49:54.576 INFO:teuthology.orchestra.run.smithi043.stdout:metadata: 2022-03-21T14:49:54.576 INFO:teuthology.orchestra.run.smithi043.stdout: name: flannel 2022-03-21T14:49:54.577 INFO:teuthology.orchestra.run.smithi043.stdout:roleRef: 2022-03-21T14:49:54.577 INFO:teuthology.orchestra.run.smithi043.stdout: apiGroup: rbac.authorization.k8s.io 2022-03-21T14:49:54.577 INFO:teuthology.orchestra.run.smithi043.stdout: kind: ClusterRole 2022-03-21T14:49:54.577 INFO:teuthology.orchestra.run.smithi043.stdout: name: flannel 2022-03-21T14:49:54.577 INFO:teuthology.orchestra.run.smithi043.stdout:subjects: 2022-03-21T14:49:54.578 INFO:teuthology.orchestra.run.smithi043.stdout:- kind: ServiceAccount 2022-03-21T14:49:54.578 INFO:teuthology.orchestra.run.smithi043.stdout: name: flannel 2022-03-21T14:49:54.578 INFO:teuthology.orchestra.run.smithi043.stdout: namespace: kube-system 2022-03-21T14:49:54.578 INFO:teuthology.orchestra.run.smithi043.stdout:--- 2022-03-21T14:49:54.579 INFO:teuthology.orchestra.run.smithi043.stdout:apiVersion: v1 2022-03-21T14:49:54.579 INFO:teuthology.orchestra.run.smithi043.stdout:kind: ServiceAccount 2022-03-21T14:49:54.579 INFO:teuthology.orchestra.run.smithi043.stdout:metadata: 2022-03-21T14:49:54.579 INFO:teuthology.orchestra.run.smithi043.stdout: name: flannel 2022-03-21T14:49:54.579 INFO:teuthology.orchestra.run.smithi043.stdout: namespace: kube-system 2022-03-21T14:49:54.580 INFO:teuthology.orchestra.run.smithi043.stdout:--- 2022-03-21T14:49:54.580 INFO:teuthology.orchestra.run.smithi043.stdout:kind: ConfigMap 2022-03-21T14:49:54.580 INFO:teuthology.orchestra.run.smithi043.stdout:apiVersion: v1 2022-03-21T14:49:54.580 INFO:teuthology.orchestra.run.smithi043.stdout:metadata: 2022-03-21T14:49:54.580 INFO:teuthology.orchestra.run.smithi043.stdout: name: kube-flannel-cfg 2022-03-21T14:49:54.581 INFO:teuthology.orchestra.run.smithi043.stdout: namespace: kube-system 2022-03-21T14:49:54.581 INFO:teuthology.orchestra.run.smithi043.stdout: labels: 2022-03-21T14:49:54.581 INFO:teuthology.orchestra.run.smithi043.stdout: tier: node 2022-03-21T14:49:54.581 INFO:teuthology.orchestra.run.smithi043.stdout: app: flannel 2022-03-21T14:49:54.582 INFO:teuthology.orchestra.run.smithi043.stdout:data: 2022-03-21T14:49:54.582 INFO:teuthology.orchestra.run.smithi043.stdout: cni-conf.json: | 2022-03-21T14:49:54.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2022-03-21T14:49:54.582 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cbr0", 2022-03-21T14:49:54.582 INFO:teuthology.orchestra.run.smithi043.stdout: "cniVersion": "0.3.1", 2022-03-21T14:49:54.583 INFO:teuthology.orchestra.run.smithi043.stdout: "plugins": [ 2022-03-21T14:49:54.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2022-03-21T14:49:54.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "flannel", 2022-03-21T14:49:54.583 INFO:teuthology.orchestra.run.smithi043.stdout: "delegate": { 2022-03-21T14:49:54.583 INFO:teuthology.orchestra.run.smithi043.stdout: "hairpinMode": true, 2022-03-21T14:49:54.584 INFO:teuthology.orchestra.run.smithi043.stdout: "isDefaultGateway": true 2022-03-21T14:49:54.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2022-03-21T14:49:54.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2022-03-21T14:49:54.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2022-03-21T14:49:54.585 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "portmap", 2022-03-21T14:49:54.585 INFO:teuthology.orchestra.run.smithi043.stdout: "capabilities": { 2022-03-21T14:49:54.585 INFO:teuthology.orchestra.run.smithi043.stdout: "portMappings": true 2022-03-21T14:49:54.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2022-03-21T14:49:54.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2022-03-21T14:49:54.586 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2022-03-21T14:49:54.586 INFO:teuthology.orchestra.run.smithi043.stdout: } 2022-03-21T14:49:54.586 INFO:teuthology.orchestra.run.smithi043.stdout: net-conf.json: | 2022-03-21T14:49:54.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2022-03-21T14:49:54.586 INFO:teuthology.orchestra.run.smithi043.stdout: "Network": "10.244.0.0/16", 2022-03-21T14:49:54.587 INFO:teuthology.orchestra.run.smithi043.stdout: "Backend": { 2022-03-21T14:49:54.587 INFO:teuthology.orchestra.run.smithi043.stdout: "Type": "vxlan" 2022-03-21T14:49:54.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2022-03-21T14:49:54.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2022-03-21T14:49:54.588 INFO:teuthology.orchestra.run.smithi043.stdout:--- 2022-03-21T14:49:54.588 INFO:teuthology.orchestra.run.smithi043.stdout:apiVersion: apps/v1 2022-03-21T14:49:54.588 INFO:teuthology.orchestra.run.smithi043.stdout:kind: DaemonSet 2022-03-21T14:49:54.588 INFO:teuthology.orchestra.run.smithi043.stdout:metadata: 2022-03-21T14:49:54.588 INFO:teuthology.orchestra.run.smithi043.stdout: name: kube-flannel-ds 2022-03-21T14:49:54.589 INFO:teuthology.orchestra.run.smithi043.stdout: namespace: kube-system 2022-03-21T14:49:54.589 INFO:teuthology.orchestra.run.smithi043.stdout: labels: 2022-03-21T14:49:54.589 INFO:teuthology.orchestra.run.smithi043.stdout: tier: node 2022-03-21T14:49:54.589 INFO:teuthology.orchestra.run.smithi043.stdout: app: flannel 2022-03-21T14:49:54.589 INFO:teuthology.orchestra.run.smithi043.stdout:spec: 2022-03-21T14:49:54.590 INFO:teuthology.orchestra.run.smithi043.stdout: selector: 2022-03-21T14:49:54.590 INFO:teuthology.orchestra.run.smithi043.stdout: matchLabels: 2022-03-21T14:49:54.590 INFO:teuthology.orchestra.run.smithi043.stdout: app: flannel 2022-03-21T14:49:54.590 INFO:teuthology.orchestra.run.smithi043.stdout: template: 2022-03-21T14:49:54.591 INFO:teuthology.orchestra.run.smithi043.stdout: metadata: 2022-03-21T14:49:54.591 INFO:teuthology.orchestra.run.smithi043.stdout: labels: 2022-03-21T14:49:54.591 INFO:teuthology.orchestra.run.smithi043.stdout: tier: node 2022-03-21T14:49:54.591 INFO:teuthology.orchestra.run.smithi043.stdout: app: flannel 2022-03-21T14:49:54.591 INFO:teuthology.orchestra.run.smithi043.stdout: spec: 2022-03-21T14:49:54.592 INFO:teuthology.orchestra.run.smithi043.stdout: affinity: 2022-03-21T14:49:54.592 INFO:teuthology.orchestra.run.smithi043.stdout: nodeAffinity: 2022-03-21T14:49:54.592 INFO:teuthology.orchestra.run.smithi043.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-03-21T14:49:54.592 INFO:teuthology.orchestra.run.smithi043.stdout: nodeSelectorTerms: 2022-03-21T14:49:54.592 INFO:teuthology.orchestra.run.smithi043.stdout: - matchExpressions: 2022-03-21T14:49:54.593 INFO:teuthology.orchestra.run.smithi043.stdout: - key: kubernetes.io/os 2022-03-21T14:49:54.593 INFO:teuthology.orchestra.run.smithi043.stdout: operator: In 2022-03-21T14:49:54.593 INFO:teuthology.orchestra.run.smithi043.stdout: values: 2022-03-21T14:49:54.593 INFO:teuthology.orchestra.run.smithi043.stdout: - linux 2022-03-21T14:49:54.593 INFO:teuthology.orchestra.run.smithi043.stdout: hostNetwork: true 2022-03-21T14:49:54.594 INFO:teuthology.orchestra.run.smithi043.stdout: priorityClassName: system-node-critical 2022-03-21T14:49:54.594 INFO:teuthology.orchestra.run.smithi043.stdout: tolerations: 2022-03-21T14:49:54.594 INFO:teuthology.orchestra.run.smithi043.stdout: - operator: Exists 2022-03-21T14:49:54.594 INFO:teuthology.orchestra.run.smithi043.stdout: effect: NoSchedule 2022-03-21T14:49:54.595 INFO:teuthology.orchestra.run.smithi043.stdout: serviceAccountName: flannel 2022-03-21T14:49:54.595 INFO:teuthology.orchestra.run.smithi043.stdout: initContainers: 2022-03-21T14:49:54.595 INFO:teuthology.orchestra.run.smithi043.stdout: - name: install-cni-plugin 2022-03-21T14:49:54.595 INFO:teuthology.orchestra.run.smithi043.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-21T14:49:54.595 INFO:teuthology.orchestra.run.smithi043.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-03-21T14:49:54.596 INFO:teuthology.orchestra.run.smithi043.stdout: command: 2022-03-21T14:49:54.596 INFO:teuthology.orchestra.run.smithi043.stdout: - cp 2022-03-21T14:49:54.596 INFO:teuthology.orchestra.run.smithi043.stdout: args: 2022-03-21T14:49:54.596 INFO:teuthology.orchestra.run.smithi043.stdout: - -f 2022-03-21T14:49:54.596 INFO:teuthology.orchestra.run.smithi043.stdout: - /flannel 2022-03-21T14:49:54.597 INFO:teuthology.orchestra.run.smithi043.stdout: - /opt/cni/bin/flannel 2022-03-21T14:49:54.597 INFO:teuthology.orchestra.run.smithi043.stdout: volumeMounts: 2022-03-21T14:49:54.597 INFO:teuthology.orchestra.run.smithi043.stdout: - name: cni-plugin 2022-03-21T14:49:54.597 INFO:teuthology.orchestra.run.smithi043.stdout: mountPath: /opt/cni/bin 2022-03-21T14:49:54.597 INFO:teuthology.orchestra.run.smithi043.stdout: - name: install-cni 2022-03-21T14:49:54.598 INFO:teuthology.orchestra.run.smithi043.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-21T14:49:54.598 INFO:teuthology.orchestra.run.smithi043.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-21T14:49:54.598 INFO:teuthology.orchestra.run.smithi043.stdout: command: 2022-03-21T14:49:54.598 INFO:teuthology.orchestra.run.smithi043.stdout: - cp 2022-03-21T14:49:54.598 INFO:teuthology.orchestra.run.smithi043.stdout: args: 2022-03-21T14:49:54.599 INFO:teuthology.orchestra.run.smithi043.stdout: - -f 2022-03-21T14:49:54.599 INFO:teuthology.orchestra.run.smithi043.stdout: - /etc/kube-flannel/cni-conf.json 2022-03-21T14:49:54.600 INFO:teuthology.orchestra.run.smithi043.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-03-21T14:49:54.600 INFO:teuthology.orchestra.run.smithi043.stdout: volumeMounts: 2022-03-21T14:49:54.600 INFO:teuthology.orchestra.run.smithi043.stdout: - name: cni 2022-03-21T14:49:54.600 INFO:teuthology.orchestra.run.smithi043.stdout: mountPath: /etc/cni/net.d 2022-03-21T14:49:54.600 INFO:teuthology.orchestra.run.smithi043.stdout: - name: flannel-cfg 2022-03-21T14:49:54.601 INFO:teuthology.orchestra.run.smithi043.stdout: mountPath: /etc/kube-flannel/ 2022-03-21T14:49:54.601 INFO:teuthology.orchestra.run.smithi043.stdout: containers: 2022-03-21T14:49:54.601 INFO:teuthology.orchestra.run.smithi043.stdout: - name: kube-flannel 2022-03-21T14:49:54.601 INFO:teuthology.orchestra.run.smithi043.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-21T14:49:54.602 INFO:teuthology.orchestra.run.smithi043.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-21T14:49:54.602 INFO:teuthology.orchestra.run.smithi043.stdout: command: 2022-03-21T14:49:54.602 INFO:teuthology.orchestra.run.smithi043.stdout: - /opt/bin/flanneld 2022-03-21T14:49:54.602 INFO:teuthology.orchestra.run.smithi043.stdout: args: 2022-03-21T14:49:54.602 INFO:teuthology.orchestra.run.smithi043.stdout: - --ip-masq 2022-03-21T14:49:54.603 INFO:teuthology.orchestra.run.smithi043.stdout: - --kube-subnet-mgr 2022-03-21T14:49:54.603 INFO:teuthology.orchestra.run.smithi043.stdout: resources: 2022-03-21T14:49:54.603 INFO:teuthology.orchestra.run.smithi043.stdout: requests: 2022-03-21T14:49:54.603 INFO:teuthology.orchestra.run.smithi043.stdout: cpu: "100m" 2022-03-21T14:49:54.603 INFO:teuthology.orchestra.run.smithi043.stdout: memory: "50Mi" 2022-03-21T14:49:54.604 INFO:teuthology.orchestra.run.smithi043.stdout: limits: 2022-03-21T14:49:54.604 INFO:teuthology.orchestra.run.smithi043.stdout: cpu: "100m" 2022-03-21T14:49:54.604 INFO:teuthology.orchestra.run.smithi043.stdout: memory: "50Mi" 2022-03-21T14:49:54.604 INFO:teuthology.orchestra.run.smithi043.stdout: securityContext: 2022-03-21T14:49:54.604 INFO:teuthology.orchestra.run.smithi043.stdout: privileged: false 2022-03-21T14:49:54.605 INFO:teuthology.orchestra.run.smithi043.stdout: capabilities: 2022-03-21T14:49:54.605 INFO:teuthology.orchestra.run.smithi043.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-03-21T14:49:54.605 INFO:teuthology.orchestra.run.smithi043.stdout: env: 2022-03-21T14:49:54.605 INFO:teuthology.orchestra.run.smithi043.stdout: - name: POD_NAME 2022-03-21T14:49:54.606 INFO:teuthology.orchestra.run.smithi043.stdout: valueFrom: 2022-03-21T14:49:54.606 INFO:teuthology.orchestra.run.smithi043.stdout: fieldRef: 2022-03-21T14:49:54.606 INFO:teuthology.orchestra.run.smithi043.stdout: fieldPath: metadata.name 2022-03-21T14:49:54.606 INFO:teuthology.orchestra.run.smithi043.stdout: - name: POD_NAMESPACE 2022-03-21T14:49:54.606 INFO:teuthology.orchestra.run.smithi043.stdout: valueFrom: 2022-03-21T14:49:54.607 INFO:teuthology.orchestra.run.smithi043.stdout: fieldRef: 2022-03-21T14:49:54.607 INFO:teuthology.orchestra.run.smithi043.stdout: fieldPath: metadata.namespace 2022-03-21T14:49:54.607 INFO:teuthology.orchestra.run.smithi043.stdout: volumeMounts: 2022-03-21T14:49:54.608 INFO:teuthology.orchestra.run.smithi043.stdout: - name: run 2022-03-21T14:49:54.608 INFO:teuthology.orchestra.run.smithi043.stdout: mountPath: /run/flannel 2022-03-21T14:49:54.608 INFO:teuthology.orchestra.run.smithi043.stdout: - name: flannel-cfg 2022-03-21T14:49:54.608 INFO:teuthology.orchestra.run.smithi043.stdout: mountPath: /etc/kube-flannel/ 2022-03-21T14:49:54.608 INFO:teuthology.orchestra.run.smithi043.stdout: - name: xtables-lock 2022-03-21T14:49:54.609 INFO:teuthology.orchestra.run.smithi043.stdout: mountPath: /run/xtables.lock 2022-03-21T14:49:54.609 INFO:teuthology.orchestra.run.smithi043.stdout: volumes: 2022-03-21T14:49:54.609 INFO:teuthology.orchestra.run.smithi043.stdout: - name: run 2022-03-21T14:49:54.609 INFO:teuthology.orchestra.run.smithi043.stdout: hostPath: 2022-03-21T14:49:54.609 INFO:teuthology.orchestra.run.smithi043.stdout: path: /run/flannel 2022-03-21T14:49:54.610 INFO:teuthology.orchestra.run.smithi043.stdout: - name: cni-plugin 2022-03-21T14:49:54.610 INFO:teuthology.orchestra.run.smithi043.stdout: hostPath: 2022-03-21T14:49:54.610 INFO:teuthology.orchestra.run.smithi043.stdout: path: /opt/cni/bin 2022-03-21T14:49:54.610 INFO:teuthology.orchestra.run.smithi043.stdout: - name: cni 2022-03-21T14:49:54.611 INFO:teuthology.orchestra.run.smithi043.stdout: hostPath: 2022-03-21T14:49:54.611 INFO:teuthology.orchestra.run.smithi043.stdout: path: /etc/cni/net.d 2022-03-21T14:49:54.611 INFO:teuthology.orchestra.run.smithi043.stdout: - name: flannel-cfg 2022-03-21T14:49:54.611 INFO:teuthology.orchestra.run.smithi043.stdout: configMap: 2022-03-21T14:49:54.611 INFO:teuthology.orchestra.run.smithi043.stdout: name: kube-flannel-cfg 2022-03-21T14:49:54.612 INFO:teuthology.orchestra.run.smithi043.stdout: - name: xtables-lock 2022-03-21T14:49:54.612 INFO:teuthology.orchestra.run.smithi043.stdout: hostPath: 2022-03-21T14:49:54.612 INFO:teuthology.orchestra.run.smithi043.stdout: path: /run/xtables.lock 2022-03-21T14:49:54.612 INFO:teuthology.orchestra.run.smithi043.stdout: type: FileOrCreate 2022-03-21T14:49:54.613 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 45903 0 --:--:-- --:--:-- --:--:-- 45903 2022-03-21T14:49:54.638 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-21T14:49:54.639 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.249.80.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-21T14:49:54.652 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.249.80.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-03-21T14:49:54.653 DEBUG:teuthology.orchestra.run.smithi043:> kubectl apply -f - 2022-03-21T14:49:57.059 INFO:teuthology.orchestra.run.smithi043.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-21T14:49:57.063 INFO:teuthology.orchestra.run.smithi043.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-03-21T14:49:57.087 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-03-21T14:49:57.095 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-03-21T14:49:57.105 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/flannel created 2022-03-21T14:49:57.114 INFO:teuthology.orchestra.run.smithi043.stdout:configmap/kube-flannel-cfg created 2022-03-21T14:49:57.168 INFO:teuthology.orchestra.run.smithi043.stdout:daemonset.apps/kube-flannel-ds created 2022-03-21T14:49:57.171 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-21T14:49:57.172 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:49:57.172 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:49:57.178 INFO:tasks.kubeadm:Scratch devices on smithi043: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-21T14:49:57.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-21T14:49:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records in 2022-03-21T14:49:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2022-03-21T14:49:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128583 s, 815 MB/s 2022-03-21T14:49:57.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-21T14:49:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records in 2022-03-21T14:49:57.228 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2022-03-21T14:49:57.228 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111994 s, 936 MB/s 2022-03-21T14:49:57.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-21T14:49:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records in 2022-03-21T14:49:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2022-03-21T14:49:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125856 s, 833 MB/s 2022-03-21T14:49:57.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-21T14:49:57.326 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records in 2022-03-21T14:49:57.326 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2022-03-21T14:49:57.327 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013347 s, 786 MB/s 2022-03-21T14:49:57.328 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:49:57.328 DEBUG:teuthology.orchestra.run.smithi065:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:49:57.336 INFO:tasks.kubeadm:Scratch devices on smithi065: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-21T14:49:57.336 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-21T14:49:57.412 INFO:teuthology.orchestra.run.smithi065.stderr:10+0 records in 2022-03-21T14:49:57.412 INFO:teuthology.orchestra.run.smithi065.stderr:10+0 records out 2022-03-21T14:49:57.412 INFO:teuthology.orchestra.run.smithi065.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01901 s, 552 MB/s 2022-03-21T14:49:57.413 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-21T14:49:57.443 INFO:teuthology.orchestra.run.smithi065.stderr:10+0 records in 2022-03-21T14:49:57.443 INFO:teuthology.orchestra.run.smithi065.stderr:10+0 records out 2022-03-21T14:49:57.443 INFO:teuthology.orchestra.run.smithi065.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0161534 s, 649 MB/s 2022-03-21T14:49:57.444 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-21T14:49:57.473 INFO:teuthology.orchestra.run.smithi065.stderr:10+0 records in 2022-03-21T14:49:57.473 INFO:teuthology.orchestra.run.smithi065.stderr:10+0 records out 2022-03-21T14:49:57.474 INFO:teuthology.orchestra.run.smithi065.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0165451 s, 634 MB/s 2022-03-21T14:49:57.475 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-21T14:49:57.554 INFO:teuthology.orchestra.run.smithi065.stderr:10+0 records in 2022-03-21T14:49:57.554 INFO:teuthology.orchestra.run.smithi065.stderr:10+0 records out 2022-03-21T14:49:57.554 INFO:teuthology.orchestra.run.smithi065.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0175831 s, 596 MB/s 2022-03-21T14:49:57.556 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:49:57.556 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:49:57.564 INFO:tasks.kubeadm:Scratch devices on smithi154: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-21T14:49:57.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-21T14:49:57.637 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records in 2022-03-21T14:49:57.637 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2022-03-21T14:49:57.637 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0166259 s, 631 MB/s 2022-03-21T14:49:57.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-21T14:49:57.667 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records in 2022-03-21T14:49:57.668 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2022-03-21T14:49:57.668 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155718 s, 673 MB/s 2022-03-21T14:49:57.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-21T14:49:57.697 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records in 2022-03-21T14:49:57.697 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2022-03-21T14:49:57.697 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0159104 s, 659 MB/s 2022-03-21T14:49:57.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-21T14:49:57.726 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records in 2022-03-21T14:49:57.726 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2022-03-21T14:49:57.727 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140981 s, 744 MB/s 2022-03-21T14:49:57.744 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-21T14:49:57.744 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi043-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi043 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi043-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi043 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi043-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi043 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi043-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi043 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi065-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi065 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi065-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi065 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi065-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi065 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi065-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi065 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi154-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi154 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi154-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi154 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi154-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi154 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi154-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi154 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-21T14:49:57.745 DEBUG:teuthology.orchestra.run.smithi043:> kubectl create -f - 2022-03-21T14:49:58.283 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi043-lv-1 created 2022-03-21T14:49:58.442 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi043-lv-2 created 2022-03-21T14:49:58.447 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi043-lv-3 created 2022-03-21T14:49:58.502 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi043-lv-4 created 2022-03-21T14:49:58.518 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi065-lv-1 created 2022-03-21T14:49:58.595 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi065-lv-2 created 2022-03-21T14:49:58.659 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi065-lv-3 created 2022-03-21T14:49:58.685 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi065-lv-4 created 2022-03-21T14:49:58.792 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi154-lv-1 created 2022-03-21T14:49:58.827 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi154-lv-2 created 2022-03-21T14:49:58.835 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi154-lv-3 created 2022-03-21T14:49:58.925 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi154-lv-4 created 2022-03-21T14:49:58.929 INFO:teuthology.orchestra.run.smithi043.stdout:storageclass.storage.k8s.io/scratch created 2022-03-21T14:49:58.935 DEBUG:teuthology.orchestra.run.smithi043:> kubectl taint node smithi043 node-role.kubernetes.io/master- || true 2022-03-21T14:49:59.002 INFO:teuthology.orchestra.run.smithi043.stdout:node/smithi043 untainted 2022-03-21T14:49:59.004 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-21T14:49:59.005 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-21T14:49:59.017 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-21T14:49:59.018 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:49:59.018 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:49:59.025 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-21T14:49:59.025 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-21T14:49:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-21T14:49:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:49:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 501 Links: 1 2022-03-21T14:49:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:49:59.033 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2022-03-21 14:49:57.201177338 +0000 2022-03-21T14:49:59.034 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2022-03-21 14:49:57.201177338 +0000 2022-03-21T14:49:59.034 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2022-03-21 14:49:57.201177338 +0000 2022-03-21T14:49:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2022-03-21T14:49:59.040 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2022-03-21T14:49:59.040 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2022-03-21T14:49:59.040 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000184345 s, 2.8 MB/s 2022-03-21T14:49:59.044 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-21T14:49:59.050 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-21T14:49:59.051 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:49:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-21T14:49:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:49:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2022-03-21 14:49:57.225177337 +0000 2022-03-21T14:49:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2022-03-21 14:49:57.225177337 +0000 2022-03-21T14:49:59.052 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2022-03-21 14:49:57.225177337 +0000 2022-03-21T14:49:59.052 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2022-03-21T14:49:59.058 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2022-03-21T14:49:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2022-03-21T14:49:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000300917 s, 1.7 MB/s 2022-03-21T14:49:59.062 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-21T14:49:59.111 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-21T14:49:59.112 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:49:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-21T14:49:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:49:59.112 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2022-03-21 14:49:57.253177336 +0000 2022-03-21T14:49:59.113 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2022-03-21 14:49:57.249177336 +0000 2022-03-21T14:49:59.113 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2022-03-21 14:49:57.249177336 +0000 2022-03-21T14:49:59.113 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2022-03-21T14:49:59.118 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2022-03-21T14:49:59.119 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2022-03-21T14:49:59.119 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000198264 s, 2.6 MB/s 2022-03-21T14:49:59.122 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-21T14:49:59.128 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-21T14:49:59.129 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:49:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-21T14:49:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:49:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2022-03-21 14:49:57.325177334 +0000 2022-03-21T14:49:59.129 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2022-03-21 14:49:57.325177334 +0000 2022-03-21T14:49:59.130 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2022-03-21 14:49:57.325177334 +0000 2022-03-21T14:49:59.130 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2022-03-21T14:49:59.135 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2022-03-21T14:49:59.135 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2022-03-21T14:49:59.135 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000250941 s, 2.0 MB/s 2022-03-21T14:49:59.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-21T14:49:59.222 INFO:teuthology.orchestra.run.smithi043.stdout:loop 2022-03-21T14:49:59.224 INFO:tasks.nvme_loop:Connecting nvme_loop smithi043:/dev/vg_nvme/lv_1... 2022-03-21T14:49:59.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-21T14:49:59.245 INFO:teuthology.orchestra.run.smithi043.stdout:1 2022-03-21T14:49:59.269 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/vg_nvme/lv_11 2022-03-21T14:49:59.285 INFO:tasks.nvme_loop:Connecting nvme_loop smithi043:/dev/vg_nvme/lv_2... 2022-03-21T14:49:59.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-21T14:49:59.346 INFO:teuthology.orchestra.run.smithi043.stdout:1 2022-03-21T14:49:59.366 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/vg_nvme/lv_21 2022-03-21T14:49:59.388 INFO:tasks.nvme_loop:Connecting nvme_loop smithi043:/dev/vg_nvme/lv_3... 2022-03-21T14:49:59.388 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-21T14:49:59.409 INFO:teuthology.orchestra.run.smithi043.stdout:1 2022-03-21T14:49:59.432 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/vg_nvme/lv_31 2022-03-21T14:49:59.455 INFO:tasks.nvme_loop:Connecting nvme_loop smithi043:/dev/vg_nvme/lv_4... 2022-03-21T14:49:59.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-21T14:49:59.479 INFO:teuthology.orchestra.run.smithi043.stdout:1 2022-03-21T14:49:59.499 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/vg_nvme/lv_41 2022-03-21T14:49:59.523 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:49:59.523 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:49:59.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo nvme list 2022-03-21T14:49:59.586 INFO:teuthology.orchestra.run.smithi043.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-21T14:49:59.586 INFO:teuthology.orchestra.run.smithi043.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-21T14:49:59.587 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/nvme0n1 PHMB751300GH480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-21T14:49:59.587 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/nvme1n1 3dc39748b2b93e86 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:49:59.587 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/nvme2n1 3b4f7e95bb271900 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:49:59.587 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/nvme3n1 0ab0fd5f188e0929 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:49:59.587 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/nvme4n1 041d48c7269a3b3f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:49:59.588 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-21T14:49:59.589 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:49:59.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/scratch_devs 2022-03-21T14:49:59.600 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:49:59.600 DEBUG:teuthology.orchestra.run.smithi065:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:49:59.609 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-21T14:49:59.609 DEBUG:teuthology.orchestra.run.smithi065:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-21T14:49:59.616 INFO:teuthology.orchestra.run.smithi065.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-21T14:49:59.616 INFO:teuthology.orchestra.run.smithi065.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:49:59.617 INFO:teuthology.orchestra.run.smithi065.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-21T14:49:59.617 INFO:teuthology.orchestra.run.smithi065.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:49:59.617 INFO:teuthology.orchestra.run.smithi065.stdout:Access: 2022-03-21 14:49:57.415232308 +0000 2022-03-21T14:49:59.617 INFO:teuthology.orchestra.run.smithi065.stdout:Modify: 2022-03-21 14:49:57.415232308 +0000 2022-03-21T14:49:59.618 INFO:teuthology.orchestra.run.smithi065.stdout:Change: 2022-03-21 14:49:57.415232308 +0000 2022-03-21T14:49:59.618 INFO:teuthology.orchestra.run.smithi065.stdout: Birth: - 2022-03-21T14:49:59.624 INFO:teuthology.orchestra.run.smithi065.stderr:1+0 records in 2022-03-21T14:49:59.624 INFO:teuthology.orchestra.run.smithi065.stderr:1+0 records out 2022-03-21T14:49:59.624 INFO:teuthology.orchestra.run.smithi065.stderr:512 bytes copied, 0.000536288 s, 955 kB/s 2022-03-21T14:49:59.627 DEBUG:teuthology.orchestra.run.smithi065:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-21T14:49:59.676 INFO:teuthology.orchestra.run.smithi065.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-21T14:49:59.676 INFO:teuthology.orchestra.run.smithi065.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:49:59.677 INFO:teuthology.orchestra.run.smithi065.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-03-21T14:49:59.677 INFO:teuthology.orchestra.run.smithi065.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:49:59.677 INFO:teuthology.orchestra.run.smithi065.stdout:Access: 2022-03-21 14:49:57.447232308 +0000 2022-03-21T14:49:59.677 INFO:teuthology.orchestra.run.smithi065.stdout:Modify: 2022-03-21 14:49:57.447232308 +0000 2022-03-21T14:49:59.678 INFO:teuthology.orchestra.run.smithi065.stdout:Change: 2022-03-21 14:49:57.447232308 +0000 2022-03-21T14:49:59.678 INFO:teuthology.orchestra.run.smithi065.stdout: Birth: - 2022-03-21T14:49:59.687 INFO:teuthology.orchestra.run.smithi065.stderr:1+0 records in 2022-03-21T14:49:59.687 INFO:teuthology.orchestra.run.smithi065.stderr:1+0 records out 2022-03-21T14:49:59.688 INFO:teuthology.orchestra.run.smithi065.stderr:512 bytes copied, 0.000531293 s, 964 kB/s 2022-03-21T14:49:59.690 DEBUG:teuthology.orchestra.run.smithi065:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-21T14:49:59.697 INFO:teuthology.orchestra.run.smithi065.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-21T14:49:59.697 INFO:teuthology.orchestra.run.smithi065.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:49:59.698 INFO:teuthology.orchestra.run.smithi065.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-03-21T14:49:59.698 INFO:teuthology.orchestra.run.smithi065.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:49:59.698 INFO:teuthology.orchestra.run.smithi065.stdout:Access: 2022-03-21 14:49:57.479232307 +0000 2022-03-21T14:49:59.698 INFO:teuthology.orchestra.run.smithi065.stdout:Modify: 2022-03-21 14:49:57.479232307 +0000 2022-03-21T14:49:59.699 INFO:teuthology.orchestra.run.smithi065.stdout:Change: 2022-03-21 14:49:57.479232307 +0000 2022-03-21T14:49:59.699 INFO:teuthology.orchestra.run.smithi065.stdout: Birth: - 2022-03-21T14:49:59.706 INFO:teuthology.orchestra.run.smithi065.stderr:1+0 records in 2022-03-21T14:49:59.706 INFO:teuthology.orchestra.run.smithi065.stderr:1+0 records out 2022-03-21T14:49:59.707 INFO:teuthology.orchestra.run.smithi065.stderr:512 bytes copied, 0.000527566 s, 970 kB/s 2022-03-21T14:49:59.711 DEBUG:teuthology.orchestra.run.smithi065:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-21T14:49:59.718 INFO:teuthology.orchestra.run.smithi065.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-21T14:49:59.718 INFO:teuthology.orchestra.run.smithi065.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:49:59.719 INFO:teuthology.orchestra.run.smithi065.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-21T14:49:59.719 INFO:teuthology.orchestra.run.smithi065.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:49:59.719 INFO:teuthology.orchestra.run.smithi065.stdout:Access: 2022-03-21 14:49:57.559232305 +0000 2022-03-21T14:49:59.719 INFO:teuthology.orchestra.run.smithi065.stdout:Modify: 2022-03-21 14:49:57.559232305 +0000 2022-03-21T14:49:59.719 INFO:teuthology.orchestra.run.smithi065.stdout:Change: 2022-03-21 14:49:57.559232305 +0000 2022-03-21T14:49:59.720 INFO:teuthology.orchestra.run.smithi065.stdout: Birth: - 2022-03-21T14:49:59.728 INFO:teuthology.orchestra.run.smithi065.stderr:1+0 records in 2022-03-21T14:49:59.728 INFO:teuthology.orchestra.run.smithi065.stderr:1+0 records out 2022-03-21T14:49:59.728 INFO:teuthology.orchestra.run.smithi065.stderr:512 bytes copied, 0.000554464 s, 923 kB/s 2022-03-21T14:49:59.732 DEBUG:teuthology.orchestra.run.smithi065:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-21T14:49:59.824 INFO:teuthology.orchestra.run.smithi065.stdout:loop 2022-03-21T14:49:59.826 INFO:tasks.nvme_loop:Connecting nvme_loop smithi065:/dev/vg_nvme/lv_1... 2022-03-21T14:49:59.827 DEBUG:teuthology.orchestra.run.smithi065:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-21T14:49:59.845 INFO:teuthology.orchestra.run.smithi065.stdout:1 2022-03-21T14:49:59.873 INFO:teuthology.orchestra.run.smithi065.stdout:/dev/vg_nvme/lv_11 2022-03-21T14:49:59.896 INFO:tasks.nvme_loop:Connecting nvme_loop smithi065:/dev/vg_nvme/lv_2... 2022-03-21T14:49:59.897 DEBUG:teuthology.orchestra.run.smithi065:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-21T14:49:59.917 INFO:teuthology.orchestra.run.smithi065.stdout:1 2022-03-21T14:49:59.942 INFO:teuthology.orchestra.run.smithi065.stdout:/dev/vg_nvme/lv_21 2022-03-21T14:49:59.968 INFO:tasks.nvme_loop:Connecting nvme_loop smithi065:/dev/vg_nvme/lv_3... 2022-03-21T14:49:59.969 DEBUG:teuthology.orchestra.run.smithi065:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-21T14:50:00.029 INFO:teuthology.orchestra.run.smithi065.stdout:1 2022-03-21T14:50:00.051 INFO:teuthology.orchestra.run.smithi065.stdout:/dev/vg_nvme/lv_31 2022-03-21T14:50:00.065 INFO:tasks.nvme_loop:Connecting nvme_loop smithi065:/dev/vg_nvme/lv_4... 2022-03-21T14:50:00.066 DEBUG:teuthology.orchestra.run.smithi065:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-21T14:50:00.083 INFO:teuthology.orchestra.run.smithi065.stdout:1 2022-03-21T14:50:00.107 INFO:teuthology.orchestra.run.smithi065.stdout:/dev/vg_nvme/lv_41 2022-03-21T14:50:00.125 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:50:00.125 DEBUG:teuthology.orchestra.run.smithi065:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:50:00.132 DEBUG:teuthology.orchestra.run.smithi065:> sudo nvme list 2022-03-21T14:50:00.187 INFO:teuthology.orchestra.run.smithi065.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-21T14:50:00.187 INFO:teuthology.orchestra.run.smithi065.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-21T14:50:00.187 INFO:teuthology.orchestra.run.smithi065.stdout:/dev/nvme0n1 CVFT53430010400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-21T14:50:00.188 INFO:teuthology.orchestra.run.smithi065.stdout:/dev/nvme1n1 72d7e98e5ddad53e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:50:00.188 INFO:teuthology.orchestra.run.smithi065.stdout:/dev/nvme2n1 c73e68959c7d7f7c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:50:00.188 INFO:teuthology.orchestra.run.smithi065.stdout:/dev/nvme3n1 c88e3fd351ec4793 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:50:00.188 INFO:teuthology.orchestra.run.smithi065.stdout:/dev/nvme4n1 eac47b9592b3abbd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:50:00.189 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-21T14:50:00.189 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:50:00.190 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd of=/scratch_devs 2022-03-21T14:50:00.202 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:50:00.202 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:50:00.209 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-21T14:50:00.210 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-21T14:50:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-21T14:50:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:50:00.218 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-21T14:50:00.218 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:50:00.218 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2022-03-21 14:49:57.642604128 +0000 2022-03-21T14:50:00.218 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2022-03-21 14:49:57.642604128 +0000 2022-03-21T14:50:00.219 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2022-03-21 14:49:57.642604128 +0000 2022-03-21T14:50:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2022-03-21T14:50:00.228 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2022-03-21T14:50:00.228 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2022-03-21T14:50:00.228 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000553528 s, 925 kB/s 2022-03-21T14:50:00.231 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-21T14:50:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-21T14:50:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:50:00.237 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-21T14:50:00.238 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:50:00.238 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2022-03-21 14:49:57.670604128 +0000 2022-03-21T14:50:00.238 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2022-03-21 14:49:57.670604128 +0000 2022-03-21T14:50:00.238 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2022-03-21 14:49:57.670604128 +0000 2022-03-21T14:50:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2022-03-21T14:50:00.243 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2022-03-21T14:50:00.243 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2022-03-21T14:50:00.243 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000415558 s, 1.2 MB/s 2022-03-21T14:50:00.247 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-21T14:50:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-21T14:50:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:50:00.253 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-21T14:50:00.254 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:50:00.254 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2022-03-21 14:49:57.702604127 +0000 2022-03-21T14:50:00.254 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2022-03-21 14:49:57.702604127 +0000 2022-03-21T14:50:00.254 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2022-03-21 14:49:57.702604127 +0000 2022-03-21T14:50:00.255 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2022-03-21T14:50:00.261 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2022-03-21T14:50:00.262 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2022-03-21T14:50:00.262 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000401175 s, 1.3 MB/s 2022-03-21T14:50:00.265 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-21T14:50:00.316 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-21T14:50:00.316 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-21T14:50:00.317 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-03-21T14:50:00.317 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-21T14:50:00.317 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2022-03-21 14:49:57.730604126 +0000 2022-03-21T14:50:00.317 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2022-03-21 14:49:57.730604126 +0000 2022-03-21T14:50:00.318 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2022-03-21 14:49:57.730604126 +0000 2022-03-21T14:50:00.318 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2022-03-21T14:50:00.322 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2022-03-21T14:50:00.323 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2022-03-21T14:50:00.323 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000515419 s, 993 kB/s 2022-03-21T14:50:00.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-21T14:50:00.695 INFO:teuthology.orchestra.run.smithi154.stdout:loop 2022-03-21T14:50:00.696 INFO:tasks.nvme_loop:Connecting nvme_loop smithi154:/dev/vg_nvme/lv_1... 2022-03-21T14:50:00.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-21T14:50:00.716 INFO:teuthology.orchestra.run.smithi154.stdout:1 2022-03-21T14:50:00.737 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/vg_nvme/lv_11 2022-03-21T14:50:00.758 INFO:tasks.nvme_loop:Connecting nvme_loop smithi154:/dev/vg_nvme/lv_2... 2022-03-21T14:50:00.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-21T14:50:00.776 INFO:teuthology.orchestra.run.smithi154.stdout:1 2022-03-21T14:50:00.801 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/vg_nvme/lv_21 2022-03-21T14:50:00.824 INFO:tasks.nvme_loop:Connecting nvme_loop smithi154:/dev/vg_nvme/lv_3... 2022-03-21T14:50:00.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-21T14:50:00.840 INFO:teuthology.orchestra.run.smithi154.stdout:1 2022-03-21T14:50:00.858 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/vg_nvme/lv_31 2022-03-21T14:50:00.873 INFO:tasks.nvme_loop:Connecting nvme_loop smithi154:/dev/vg_nvme/lv_4... 2022-03-21T14:50:00.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-21T14:50:00.934 INFO:teuthology.orchestra.run.smithi154.stdout:1 2022-03-21T14:50:00.960 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/vg_nvme/lv_41 2022-03-21T14:50:00.979 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:50:00.980 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:50:00.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo nvme list 2022-03-21T14:50:00.996 INFO:teuthology.orchestra.run.smithi154.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-21T14:50:00.997 INFO:teuthology.orchestra.run.smithi154.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-21T14:50:00.997 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/nvme0n1 CVFT6233003B400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-21T14:50:00.997 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/nvme1n1 3e853659da982200 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:50:00.997 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/nvme2n1 2a606a0cd0489be0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:50:00.998 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/nvme3n1 18b41c475d8125d4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:50:00.998 INFO:teuthology.orchestra.run.smithi154.stdout:/dev/nvme4n1 073217f2dbc0e965 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-21T14:50:00.999 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-21T14:50:00.999 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:50:00.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/scratch_devs 2022-03-21T14:50:01.094 INFO:teuthology.run_tasks:Running task rook... 2022-03-21T14:50:01.164 INFO:tasks.rook:Rook start 2022-03-21T14:50:01.164 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e', 'rook_image': 'rook/ceph:master'} 2022-03-21T14:50:01.165 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-21T14:50:01.165 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-21T14:50:01.165 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-21T14:50:01.202 INFO:teuthology.orchestra.run.smithi043.stderr:Cloning into 'rook'... 2022-03-21T14:50:03.732 INFO:tasks.rook:/home/teuthworker 2022-03-21T14:50:03.733 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/sbin:/usr/sbin ip addr show 2022-03-21T14:50:03.738 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-21T14:50:03.738 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-21T14:50:03.738 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2022-03-21T14:50:03.738 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2022-03-21T14:50:03.739 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2022-03-21T14:50:03.739 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2022-03-21T14:50:03.739 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-21T14:50:03.739 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2022-03-21T14:50:03.740 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2022-03-21T14:50:03.740 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-21T14:50:03.740 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2022-03-21T14:50:03.740 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2022-03-21T14:50:03.741 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-21T14:50:03.741 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2022-03-21T14:50:03.741 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2022-03-21T14:50:03.741 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-21T14:50:03.741 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 42828sec preferred_lft 42828sec 2022-03-21T14:50:03.742 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link 2022-03-21T14:50:03.742 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2022-03-21T14:50:03.742 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-21T14:50:03.742 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2022-03-21T14:50:03.743 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2022-03-21T14:50:03.743 INFO:teuthology.orchestra.run.smithi043.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-21T14:50:03.743 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 02:42:4f:8d:52:88 brd ff:ff:ff:ff:ff:ff 2022-03-21T14:50:03.743 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-21T14:50:03.743 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2022-03-21T14:50:03.744 DEBUG:teuthology.orchestra.run.smithi043:> which systemctl 2022-03-21T14:50:03.786 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/systemctl 2022-03-21T14:50:03.787 DEBUG:teuthology.orchestra.run.smithi043:> true 2022-03-21T14:50:03.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo lsb_release -is 2022-03-21T14:50:03.868 INFO:teuthology.orchestra.run.smithi043.stdout:Ubuntu 2022-03-21T14:50:03.874 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-21T14:50:03.874 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-21T14:50:03.890 DEBUG:teuthology.orchestra.remote:smithi043:rook/deploy/examples/operator.yaml is 20KB 2022-03-21T14:50:03.905 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:50:03.905 DEBUG:teuthology.orchestra.run.smithi043:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-21T14:50:03.914 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-21T14:50:03.957 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:50:03.958 DEBUG:teuthology.orchestra.run.smithi043:> dd of=operator.yaml 2022-03-21T14:50:03.963 INFO:tasks.rook:Deploying operator 2022-03-21T14:50:03.973 DEBUG:teuthology.orchestra.remote:smithi043:rook/deploy/examples/operator.yaml is 20KB 2022-03-21T14:50:03.999 DEBUG:teuthology.orchestra.remote:smithi043:rook/deploy/examples/operator.yaml is 20KB 2022-03-21T14:50:04.014 DEBUG:teuthology.orchestra.run.smithi043:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-21T14:50:04.235 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-21T14:50:04.282 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-21T14:50:04.408 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-21T14:50:04.438 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-21T14:50:04.624 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-21T14:50:04.692 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-21T14:50:04.782 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-21T14:50:04.794 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-21T14:50:04.853 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-21T14:50:04.955 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-21T14:50:05.162 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-21T14:50:05.209 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-21T14:50:05.333 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-21T14:50:05.401 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-21T14:50:05.506 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-21T14:50:05.520 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-21T14:50:05.569 INFO:teuthology.orchestra.run.smithi043.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-21T14:50:05.694 INFO:teuthology.orchestra.run.smithi043.stdout:namespace/rook-ceph created 2022-03-21T14:50:05.736 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-21T14:50:05.761 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-21T14:50:05.853 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-21T14:50:05.888 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-21T14:50:05.920 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-21T14:50:05.947 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-21T14:50:05.950 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-21T14:50:05.999 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-21T14:50:06.002 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-21T14:50:06.009 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-21T14:50:06.016 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-21T14:50:06.025 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-21T14:50:06.033 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-21T14:50:06.087 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-21T14:50:06.137 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-21T14:50:06.141 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-21T14:50:06.167 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-21T14:50:06.187 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-21T14:50:06.204 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-21T14:50:06.229 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-21T14:50:06.292 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-21T14:50:06.295 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-21T14:50:06.313 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-21T14:50:06.324 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-21T14:50:06.350 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-21T14:50:06.379 INFO:teuthology.orchestra.run.smithi043.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-21T14:50:06.465 INFO:teuthology.orchestra.run.smithi043.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-21T14:50:06.465 INFO:teuthology.orchestra.run.smithi043.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-21T14:50:06.468 INFO:teuthology.orchestra.run.smithi043.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-21T14:50:06.500 INFO:teuthology.orchestra.run.smithi043.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-21T14:50:06.525 INFO:teuthology.orchestra.run.smithi043.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-21T14:50:06.623 INFO:teuthology.orchestra.run.smithi043.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-21T14:50:06.649 INFO:teuthology.orchestra.run.smithi043.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-21T14:50:06.690 INFO:teuthology.orchestra.run.smithi043.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-21T14:50:06.715 INFO:teuthology.orchestra.run.smithi043.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-21T14:50:06.791 INFO:teuthology.orchestra.run.smithi043.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-21T14:50:06.794 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-21T14:50:06.841 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-21T14:50:06.850 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-21T14:50:06.858 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-21T14:50:06.866 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-21T14:50:06.875 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-21T14:50:06.951 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-21T14:50:06.958 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-21T14:50:06.984 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-21T14:50:07.009 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-21T14:50:07.025 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-21T14:50:07.109 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-21T14:50:07.113 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-21T14:50:07.152 INFO:teuthology.orchestra.run.smithi043.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-21T14:50:07.167 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-21T14:50:07.257 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/rook-ceph-mgr created 2022-03-21T14:50:07.262 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/rook-ceph-osd created 2022-03-21T14:50:07.279 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-21T14:50:07.307 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/rook-ceph-system created 2022-03-21T14:50:07.329 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-21T14:50:07.416 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-21T14:50:07.439 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-21T14:50:07.514 INFO:teuthology.orchestra.run.smithi043.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-21T14:50:07.574 INFO:teuthology.orchestra.run.smithi043.stdout:configmap/rook-ceph-operator-config created 2022-03-21T14:50:07.615 INFO:teuthology.orchestra.run.smithi043.stdout:deployment.apps/rook-ceph-operator created 2022-03-21T14:50:07.619 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-21T14:50:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:No resources found in rook-ceph namespace. 2022-03-21T14:50:17.797 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-21T14:50:17.865 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:50:17.865 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-operator-75dd789779-jchcs 0/1 ContainerCreating 0 10s 2022-03-21T14:50:27.867 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-21T14:50:27.938 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:50:27.938 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-operator-75dd789779-jchcs 0/1 ContainerCreating 0 20s 2022-03-21T14:50:37.941 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-21T14:50:38.000 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:50:38.001 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-operator-75dd789779-jchcs 0/1 ContainerCreating 0 30s 2022-03-21T14:50:48.003 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-21T14:50:48.077 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:50:48.077 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-operator-75dd789779-jchcs 1/1 Running 0 41s 2022-03-21T14:50:48.079 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-jchcs 2022-03-21T14:50:48.088 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:50:48.088 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:50:48.136 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:50:48.137 DEBUG:teuthology.orchestra.run.smithi065:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:50:48.145 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:50:48.145 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2022-03-21T14:50:48.154 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-21T14:50:48.154 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-21T14:50:48.155 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-21T14:50:48.155 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-21T14:50:48.155 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-21T14:50:48.156 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-21T14:50:48.156 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-21T14:50:48.156 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-21T14:50:48.159 DEBUG:teuthology.orchestra.run.smithi043:> kubectl create -f - 2022-03-21T14:50:48.174 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.153314 I | rookcmd: starting Rook v1.8.0-alpha.0.443.g833c458f6 with arguments '/usr/local/bin/rook ceph operator' 2022-03-21T14:50:48.178 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.153423 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-21T14:50:48.178 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.153427 I | cephcmd: starting Rook-Ceph operator 2022-03-21T14:50:48.178 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.306289 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-21T14:50:48.179 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.320515 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-21T14:50:48.179 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.320560 I | operator: watching all namespaces for Ceph CRs 2022-03-21T14:50:48.179 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.320653 I | operator: setting up schemes 2022-03-21T14:50:48.179 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.323870 I | operator: setting up the controller-runtime manager 2022-03-21T14:50:48.179 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.528306 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-21T14:50:48.180 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.531551 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-21T14:50:48.180 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.531630 I | ceph-cluster-controller: successfully started 2022-03-21T14:50:48.180 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.531722 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-21T14:50:48.180 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.531755 I | ceph-crashcollector-controller: successfully started 2022-03-21T14:50:48.181 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.531786 I | ceph-block-pool-controller: successfully started 2022-03-21T14:50:48.181 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.531840 I | ceph-object-store-user-controller: successfully started 2022-03-21T14:50:48.181 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.531876 I | ceph-object-realm-controller: successfully started 2022-03-21T14:50:48.181 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.531899 I | ceph-object-zonegroup-controller: successfully started 2022-03-21T14:50:48.181 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.531936 I | ceph-object-zone-controller: successfully started 2022-03-21T14:50:48.182 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532253 I | ceph-object-controller: successfully started 2022-03-21T14:50:48.182 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532314 I | ceph-file-controller: successfully started 2022-03-21T14:50:48.182 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532378 I | ceph-nfs-controller: successfully started 2022-03-21T14:50:48.182 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532419 I | ceph-rbd-mirror-controller: successfully started 2022-03-21T14:50:48.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532451 I | ceph-client-controller: successfully started 2022-03-21T14:50:48.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532477 I | ceph-filesystem-mirror-controller: successfully started 2022-03-21T14:50:48.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532510 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-21T14:50:48.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532534 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-21T14:50:48.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532569 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-21T14:50:48.184 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532593 I | ceph-bucket-topic: successfully started 2022-03-21T14:50:48.184 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532611 I | ceph-bucket-notification: successfully started 2022-03-21T14:50:48.184 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532629 I | ceph-bucket-notification: successfully started 2022-03-21T14:50:48.184 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.532646 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-21T14:50:48.185 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.534033 I | operator: starting the controller-runtime manager 2022-03-21T14:50:48.185 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.640065 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-21T14:50:48.185 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.640123 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-21T14:50:48.185 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.640147 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-21T14:50:48.185 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.643091 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-21T14:50:48.186 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.643140 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-21T14:50:48.186 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.645096 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-21T14:50:48.186 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.645676 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-21T14:50:48.186 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.651819 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-21T14:50:48.187 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.651844 I | ceph-csi: CSI CephFS driver disabled 2022-03-21T14:50:48.187 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.651856 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-21T14:50:48.187 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.653932 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-21T14:50:48.187 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:45.660335 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-21T14:50:49.001 INFO:teuthology.orchestra.run.smithi043.stdout:configmap/rook-config-override created 2022-03-21T14:50:49.006 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-21T14:50:49.007 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:50:49.007 DEBUG:teuthology.orchestra.run.smithi043:> dd of=cluster.yaml 2022-03-21T14:50:49.014 DEBUG:teuthology.orchestra.run.smithi043:> kubectl create -f cluster.yaml 2022-03-21T14:50:51.438 INFO:teuthology.orchestra.run.smithi043.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-21T14:50:51.446 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.448214 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-21T14:50:51.455 DEBUG:teuthology.orchestra.remote:smithi043:rook/deploy/examples/operator.yaml is 20KB 2022-03-21T14:50:51.467 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.470044 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:50:51.469 DEBUG:teuthology.orchestra.run.smithi043:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-21T14:50:51.488 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.491408 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-21T14:50:51.523 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.525895 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-21T14:50:51.527 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.526902 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-21T14:50:51.554 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556637 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-21T14:50:51.554 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556668 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-21T14:50:51.555 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556677 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-21T14:50:51.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556684 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-21T14:50:51.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556693 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-21T14:50:51.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556701 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-21T14:50:51.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556710 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-21T14:50:51.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556720 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-21T14:50:51.557 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556729 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-21T14:50:51.557 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556737 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-21T14:50:51.557 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556744 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-21T14:50:51.557 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556753 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-21T14:50:51.557 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556766 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-21T14:50:51.558 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556780 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-21T14:50:51.558 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556791 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-21T14:50:51.558 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556805 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-21T14:50:51.558 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556892 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:50:51.558 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:50:51.556920 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:50:51.766 INFO:teuthology.orchestra.run.smithi043.stdout:deployment.apps/rook-ceph-tools created 2022-03-21T14:50:51.769 INFO:tasks.rook:Waiting for tools container to start 2022-03-21T14:50:51.770 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:50:51.878 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:50:51.878 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 Pending 0 0s 2022-03-21T14:50:56.884 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:50:56.968 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:50:56.968 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 5s 2022-03-21T14:51:01.970 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:02.042 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:02.042 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 11s 2022-03-21T14:51:07.045 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:07.115 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:07.116 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 16s 2022-03-21T14:51:12.118 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:12.189 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:12.190 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 21s 2022-03-21T14:51:17.193 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:17.263 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:17.263 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 26s 2022-03-21T14:51:22.266 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:22.333 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:22.334 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 31s 2022-03-21T14:51:27.336 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:27.405 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:27.405 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 36s 2022-03-21T14:51:32.408 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:32.469 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:32.469 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 41s 2022-03-21T14:51:37.473 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:37.540 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:37.540 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 46s 2022-03-21T14:51:42.542 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:42.605 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:42.606 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 51s 2022-03-21T14:51:47.608 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:47.674 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:47.674 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 56s 2022-03-21T14:51:52.678 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:52.740 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:52.741 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 61s 2022-03-21T14:51:57.744 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:51:57.811 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:51:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 66s 2022-03-21T14:52:02.815 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:02.884 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:02.885 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 71s 2022-03-21T14:52:07.887 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:07.949 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:07.950 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 76s 2022-03-21T14:52:11.125 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.125199 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-21T14:52:11.125 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.125229 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-21T14:52:11.157 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.157823 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-21T14:52:11.278 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.277736 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-21 14:52:11.265373521 +0000 UTC m=+86.133029334 LastTransitionTime:2022-03-21 14:52:11.265373408 +0000 UTC m=+86.133029231}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T14:52:11.326 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326254 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-21T14:52:11.326 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326288 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-21T14:52:11.327 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326299 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-21T14:52:11.329 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326305 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-21T14:52:11.329 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326311 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-21T14:52:11.329 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326317 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-21T14:52:11.329 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326323 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-21T14:52:11.330 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326331 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-21T14:52:11.330 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326343 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-21T14:52:11.330 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326353 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-21T14:52:11.330 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326359 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-21T14:52:11.331 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326366 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-21T14:52:11.331 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326372 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-21T14:52:11.331 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326378 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-21T14:52:11.331 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326384 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-21T14:52:11.331 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326391 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-21T14:52:11.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326397 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-21T14:52:11.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326403 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-21T14:52:11.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326409 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-21T14:52:11.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326419 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-21T14:52:11.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326431 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-21T14:52:11.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326438 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-21T14:52:11.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326444 I | ceph-csi: Kubernetes version is 1.23 2022-03-21T14:52:11.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326452 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-21T14:52:11.334 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.326458 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-21T14:52:11.390 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.389530 I | op-mon: start running mons 2022-03-21T14:52:11.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.392646 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-21T14:52:11.401 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.401065 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:52:11.401 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.401085 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:52:11.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.401090 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:52:11.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.401093 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:52:11.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.401097 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:52:11.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.401100 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:52:11.403 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.401104 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-21T14:52:11.443 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.443231 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:52:11.444 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.443271 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:52:11.444 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.443280 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-21T14:52:11.494 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.493693 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-21T14:52:11.506 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.506279 I | op-mon: creating mon secrets for a new cluster 2022-03-21T14:52:11.592 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.592113 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:52:11.592 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.592150 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:52:11.592 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.592160 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-21T14:52:11.745 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.744928 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:52:11.745 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.744950 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:52:11.746 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.744955 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-21T14:52:11.824 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:11.824657 I | ceph-csi: successfully started CSI CephFS driver 2022-03-21T14:52:12.259 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:12.258941 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-21T14:52:12.455 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:12.454677 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-21T14:52:12.456 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:12.454712 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-21T14:52:12.459 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:12.457457 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-21T14:52:12.459 INFO:tasks.rook.operator.smithi043.stdout:I0321 14:52:12.457573 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-21T14:52:12.539 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:12.539780 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-21T14:52:12.833 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:12.833264 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-21T14:52:12.837 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:12.837140 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-21T14:52:12.837 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:12.837159 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-21T14:52:12.952 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:13.225 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:13.225224 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-21T14:52:13.229 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:13.229 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 82s 2022-03-21T14:52:13.484 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:13.484894 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:52:13.485 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:13.485147 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:52:15.062 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:15.062632 I | op-mon: targeting the mon count 3 2022-03-21T14:52:15.072 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:15.072118 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-21T14:52:15.105 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:15.105171 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-21T14:52:15.241 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:15.240462 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-21T14:52:15.851 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:15.851470 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi154 2022-03-21T14:52:15.851 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:15.851502 I | op-mon: mon a assigned to node smithi154 2022-03-21T14:52:16.213 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:16.213361 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi065 2022-03-21T14:52:16.213 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:16.213395 I | op-mon: mon b assigned to node smithi065 2022-03-21T14:52:16.424 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:16.424931 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi043 2022-03-21T14:52:16.425 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:16.424953 I | op-mon: mon c assigned to node smithi043 2022-03-21T14:52:16.433 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:16.430594 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-21T14:52:16.436 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:16.434448 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-21T14:52:16.777 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:16.777273 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-21T14:52:17.035 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:17.035419 I | op-mon: creating mon a 2022-03-21T14:52:17.395 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:17.395164 I | op-mon: mon "a" ip is 10.97.100.133 2022-03-21T14:52:17.946 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:17.946737 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-21T14:52:17.947 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:17.946851 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-21T14:52:17.947 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:17.947020 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789"]}] data:a=10.97.100.133:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:-1] 2022-03-21T14:52:18.232 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:18.375 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:18.375285 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:52:18.378 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:18.378 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 87s 2022-03-21T14:52:18.438 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:18.438520 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:52:18.799 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:18.799632 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-21T14:52:19.206 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:19.205965 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-21T14:52:19.806 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:19.806864 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789"]}] data:a=10.97.100.133:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:0] 2022-03-21T14:52:19.807 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:19.806890 I | op-mon: waiting for mon quorum with [a] 2022-03-21T14:52:19.991 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:19.991754 I | op-mon: mon a is not yet running 2022-03-21T14:52:19.991 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:19.991778 I | op-mon: mons running: [] 2022-03-21T14:52:23.380 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:23.459 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:23.459 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 92s 2022-03-21T14:52:28.461 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:28.529 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:28.529 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 97s 2022-03-21T14:52:33.531 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:33.598 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:33.598 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 102s 2022-03-21T14:52:38.601 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:38.679 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:38.679 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 107s 2022-03-21T14:52:40.213 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:40.213847 I | op-mon: mons running: [a] 2022-03-21T14:52:43.683 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:44.081 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:44.081 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 113s 2022-03-21T14:52:49.083 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:49.350 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:49.350 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 118s 2022-03-21T14:52:50.677 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:50.677883 I | op-mon: Monitors in quorum: [a] 2022-03-21T14:52:50.677 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:50.677915 I | op-mon: mons created: 1 2022-03-21T14:52:50.900 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:50.900930 I | op-mon: waiting for mon quorum with [a] 2022-03-21T14:52:51.048 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:51.049058 I | op-mon: mons running: [a] 2022-03-21T14:52:51.280 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:51.281159 I | op-mon: Monitors in quorum: [a] 2022-03-21T14:52:51.280 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:51.281202 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:52:51.565 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:51.566128 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:52:51.565 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:51.566167 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:52:51.833 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:51.833697 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:52:51.833 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:51.833719 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:52:52.116 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:52.116954 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:52:52.116 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:52.116978 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:52:52.925 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:52.926335 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:52:52.926 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:52.926362 I | op-config: deleting "log file" option from the mon configuration database 2022-03-21T14:52:53.165 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:53.166519 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-21T14:52:53.166 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:53.166552 I | op-mon: creating mon b 2022-03-21T14:52:54.353 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:52:54.812 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:52:54.812 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 2m3s 2022-03-21T14:52:55.841 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:55.841958 I | op-mon: mon "a" ip is 10.97.100.133 2022-03-21T14:52:57.090 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:57.088515 I | op-mon: mon "b" ip is 10.96.127.40 2022-03-21T14:52:57.874 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:57.874794 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:0] 2022-03-21T14:52:58.606 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:58.607161 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:52:58.606 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:52:58.607333 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:52:59.815 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:53:00.865 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:53:00.865 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 2m9s 2022-03-21T14:53:01.168 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:01.169339 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-21T14:53:02.400 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:02.401702 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-21T14:53:02.401 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:02.401742 I | op-mon: waiting for mon quorum with [a b] 2022-03-21T14:53:02.796 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:02.797249 I | op-mon: mon b is not yet running 2022-03-21T14:53:02.796 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:02.797283 I | op-mon: mons running: [a] 2022-03-21T14:53:03.043 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:03.044644 I | op-mon: Monitors in quorum: [a] 2022-03-21T14:53:05.432 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:05.433316 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-21T14:53:05.868 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:53:07.461 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:53:07.462 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 0/1 ContainerCreating 0 2m16s 2022-03-21T14:53:09.404 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:09.405531 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:1] 2022-03-21T14:53:09.405 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:09.405570 I | op-mon: waiting for mon quorum with [a b] 2022-03-21T14:53:09.751 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:09.752216 I | op-mon: mon b is not yet running 2022-03-21T14:53:09.751 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:09.752240 I | op-mon: mons running: [a] 2022-03-21T14:53:09.979 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:09.980578 I | op-mon: Monitors in quorum: [a] 2022-03-21T14:53:09.980 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:09.980613 I | op-mon: mons created: 2 2022-03-21T14:53:10.184 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:10.185221 I | op-mon: waiting for mon quorum with [a b] 2022-03-21T14:53:10.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:10.197099 I | op-mon: mon b is not yet running 2022-03-21T14:53:10.196 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:10.197111 I | op-mon: mons running: [a] 2022-03-21T14:53:12.466 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-21T14:53:12.643 INFO:teuthology.orchestra.run.smithi043.stdout:NAME READY STATUS RESTARTS AGE 2022-03-21T14:53:12.643 INFO:teuthology.orchestra.run.smithi043.stdout:rook-ceph-tools-d6d7c985c-8rlv4 1/1 Running 0 2m21s 2022-03-21T14:53:12.644 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-21T14:53:12.644 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:53:13.225 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:53:13.225 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:53:13.226 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:53:13.226 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:53:13.226 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:53:13.226 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:53:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:53:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:53:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:53:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:53:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:53:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:53:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:53:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:53:13.233 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:53:15.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:15.210903 I | op-mon: mon b is not yet running 2022-03-21T14:53:15.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:15.210933 I | op-mon: mons running: [a] 2022-03-21T14:53:20.225 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:20.226556 I | op-mon: mon b is not yet running 2022-03-21T14:53:20.225 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:20.226580 I | op-mon: mons running: [a] 2022-03-21T14:53:23.234 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:53:23.760 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:53:23.761 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:53:23.761 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:53:23.761 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:53:23.761 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:53:23.762 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:53:23.762 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:53:23.762 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:53:23.762 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:53:23.762 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:53:23.763 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:53:23.763 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:53:23.763 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:53:23.765 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:53:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:53:25.240 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:25.241804 I | op-mon: mon b is not yet running 2022-03-21T14:53:25.240 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:25.241834 I | op-mon: mons running: [a] 2022-03-21T14:53:30.255 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:30.257409 I | op-mon: mon b is not yet running 2022-03-21T14:53:30.256 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:30.257432 I | op-mon: mons running: [a] 2022-03-21T14:53:33.770 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:53:34.260 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:53:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:53:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:53:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:53:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:53:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:53:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:53:34.263 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:53:34.263 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:53:34.263 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:53:34.263 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:53:34.264 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:53:34.264 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:53:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:53:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:53:35.313 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:35.314265 I | op-mon: mon b is not yet running 2022-03-21T14:53:35.313 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:35.314288 I | op-mon: mons running: [a] 2022-03-21T14:53:40.327 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:40.329129 I | op-mon: mon b is not yet running 2022-03-21T14:53:40.327 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:40.329161 I | op-mon: mons running: [a] 2022-03-21T14:53:44.269 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:53:44.770 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:53:44.770 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:53:44.771 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:53:44.771 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:53:44.771 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:53:44.771 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:53:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:53:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:53:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:53:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:53:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:53:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:53:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:53:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:53:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:53:45.363 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:45.352526 I | op-mon: mons running: [a b] 2022-03-21T14:53:51.608 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:51.610911 I | op-mon: Monitors in quorum: [a b] 2022-03-21T14:53:51.609 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:51.610945 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:53:51.847 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:51.849855 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:53:51.848 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:51.849886 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:53:52.060 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:52.062297 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:53:52.060 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:52.062328 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:53:52.290 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:52.291853 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:53:52.290 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:52.291888 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:53:52.508 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:52.509846 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:53:52.508 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:52.509872 I | op-config: deleting "log file" option from the mon configuration database 2022-03-21T14:53:52.775 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:52.776876 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-21T14:53:52.775 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:52.776911 I | op-mon: creating mon c 2022-03-21T14:53:52.923 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:52.923116 I | op-mon: mon "a" ip is 10.97.100.133 2022-03-21T14:53:53.044 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:53.046212 I | op-mon: mon "b" ip is 10.96.127.40 2022-03-21T14:53:53.079 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:53.081086 I | op-mon: mon "c" ip is 10.111.185.43 2022-03-21T14:53:53.212 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:53.214760 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789","10.111.185.43:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:1] 2022-03-21T14:53:53.778 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:53.780648 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:53:53.778 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:53.780787 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:53:54.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:54.211698 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-03-21T14:53:54.215 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:54.218241 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-21T14:53:54.225 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:54.227645 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-21T14:53:54.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:54.232432 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-21T14:53:54.239 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:54.241802 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-21T14:53:54.422 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:54.424379 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-21T14:53:54.779 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:53:55.182 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:55.184152 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.96.127.40:6789","10.111.185.43:6789","10.97.100.133:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:53:55.182 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:55.184185 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:53:55.356 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:53:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:53:55.357 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:53:55.357 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:53:55.357 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:53:55.357 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:53:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:53:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:53:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:53:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:53:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:53:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:53:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:53:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:53:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:53:55.787 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:55.789037 I | op-mon: mon c is not yet running 2022-03-21T14:53:55.787 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:55.789076 I | op-mon: mons running: [a b] 2022-03-21T14:53:56.031 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:56.033517 I | op-mon: Monitors in quorum: [a b] 2022-03-21T14:53:56.031 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:56.033550 I | op-mon: mons created: 3 2022-03-21T14:53:56.251 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:56.253530 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:53:56.383 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:56.385618 I | op-mon: mon c is not yet running 2022-03-21T14:53:56.383 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:53:56.385655 I | op-mon: mons running: [a b] 2022-03-21T14:54:01.421 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:01.423942 I | op-mon: mon c is not yet running 2022-03-21T14:54:01.422 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:01.423976 I | op-mon: mons running: [a b] 2022-03-21T14:54:05.365 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:54:05.867 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:54:05.867 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:54:05.868 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:54:05.868 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:54:05.868 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:54:05.868 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:54:05.869 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:54:05.869 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:54:05.869 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:54:05.869 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:54:05.869 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:54:05.870 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:54:05.870 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:54:05.872 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:54:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:54:06.444 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:06.446579 I | op-mon: mon c is not yet running 2022-03-21T14:54:06.444 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:06.446616 I | op-mon: mons running: [a b] 2022-03-21T14:54:11.480 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:11.482494 I | op-mon: mon c is not yet running 2022-03-21T14:54:11.480 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:11.482529 I | op-mon: mons running: [a b] 2022-03-21T14:54:15.876 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:54:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:54:16.360 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:54:16.361 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:54:16.361 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:54:16.361 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:54:16.361 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:54:16.362 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:54:16.362 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:54:16.362 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:54:16.362 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:54:16.362 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:54:16.363 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:54:16.363 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:54:16.364 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:54:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:54:16.512 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:16.510999 I | op-mon: mon c is not yet running 2022-03-21T14:54:16.512 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:16.511041 I | op-mon: mons running: [a b] 2022-03-21T14:54:21.532 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:21.534904 I | op-mon: mon c is not yet running 2022-03-21T14:54:21.533 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:21.534937 I | op-mon: mons running: [a b] 2022-03-21T14:54:26.367 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:54:26.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:26.774662 I | op-mon: mon c is not yet running 2022-03-21T14:54:26.773 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:26.774702 I | op-mon: mons running: [a b] 2022-03-21T14:54:27.159 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:54:27.159 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:54:27.160 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:54:27.160 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:54:27.160 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:54:27.160 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:54:27.161 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:54:27.161 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:54:27.161 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:54:27.161 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:54:27.161 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:54:27.162 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:54:27.162 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:54:27.163 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:54:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:54:31.801 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:31.804299 I | op-mon: mon c is not yet running 2022-03-21T14:54:31.801 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:31.804338 I | op-mon: mons running: [a b] 2022-03-21T14:54:36.824 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:36.827487 I | op-mon: mons running: [a b c] 2022-03-21T14:54:37.168 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:54:38.879 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:54:38.879 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:54:38.880 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:54:38.880 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:54:38.880 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:54:38.880 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:54:38.880 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:54:38.881 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:54:38.881 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:54:38.881 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:54:38.881 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:54:38.881 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:54:38.882 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:54:38.884 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:54:40.066 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.069176 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:54:40.066 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.069235 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:54:40.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.290158 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:54:40.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.290189 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:54:40.493 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.496922 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:54:40.494 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.496956 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:54:40.723 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.726683 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:54:40.724 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.726704 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:54:40.949 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.952264 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:54:40.949 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:40.952295 I | op-config: deleting "log file" option from the mon configuration database 2022-03-21T14:54:41.189 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:41.192246 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-21T14:54:41.189 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:41.192292 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-21T14:54:41.588 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:41.591585 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-21T14:54:41.937 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:41.940548 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-21T14:54:42.360 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:42.363262 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-21T14:54:42.790 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:42.793495 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-21T14:54:42.790 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:42.793532 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-21T14:54:43.152 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:43.155452 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-21T14:54:43.152 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:43.155475 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-21T14:54:43.153 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:43.155479 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-21T14:54:43.562 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:43.565303 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-21T14:54:43.577 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:43.580701 I | op-mgr: start running mgr 2022-03-21T14:54:43.578 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:43.580743 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-21T14:54:44.268 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:44.271104 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi043": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T14:54:48.886 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:54:49.754 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:54:49.755 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:54:49.755 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:54:49.755 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:54:49.755 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:54:49.756 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:54:49.756 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:54:49.756 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:54:49.756 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:54:49.757 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:54:49.757 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:54:49.757 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:54:49.757 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:54:49.759 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:54:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:54:50.463 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:54:50.466628 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi154": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T14:54:59.763 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:55:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:55:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:no valid command found; 10 closest matches: 2022-03-21T14:55:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:pg stat 2022-03-21T14:55:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:pg getmap 2022-03-21T14:55:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-21T14:55:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-21T14:55:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_pools_json 2022-03-21T14:55:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-pool [...] 2022-03-21T14:55:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-primary [] [...] 2022-03-21T14:55:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls-by-osd [] [...] 2022-03-21T14:55:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:pg ls [] [...] 2022-03-21T14:55:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-21T14:55:00.256 INFO:teuthology.orchestra.run.smithi043.stderr:Error EINVAL: invalid command 2022-03-21T14:55:00.257 INFO:teuthology.orchestra.run.smithi043.stderr:command terminated with exit code 22 2022-03-21T14:55:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-21T14:55:05.403 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.406844 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-21T14:55:05.408 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.412255 I | op-mgr: setting services to point to mgr "a" 2022-03-21T14:55:05.458 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.462825 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-21T14:55:05.463 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.462939 I | op-mgr: successful modules: balancer 2022-03-21T14:55:05.467 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.470232 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-21T14:55:05.468 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.470263 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-21T14:55:05.468 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.470273 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-21T14:55:05.536 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.540567 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-21T14:55:05.540 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.544069 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-21T14:55:05.541 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.544181 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-21T14:55:05.541 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:05.544192 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-21T14:55:06.016 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:06.019704 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-21T14:55:06.016 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:06.019736 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:55:06.028 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:06.032378 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-21T14:55:06.028 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:06.032411 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-21T14:55:06.029 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:06.032435 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-21T14:55:06.087 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:06.091427 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-21T14:55:06.091 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:06.093716 I | op-mgr: successful modules: dashboard 2022-03-21T14:55:06.099 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:06.097023 I | op-mgr: successful modules: prometheus 2022-03-21T14:55:10.262 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status -f json 2022-03-21T14:55:14.987 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:14.991083 I | op-mgr: successful modules: orchestrator modules 2022-03-21T14:55:15.040 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:55:15.040 INFO:teuthology.orchestra.run.smithi043.stdout:{"available": true, "backend": "rook"} 2022-03-21T14:55:15.070 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-21T14:55:15.071 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-21T14:55:15.743 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch apply osd --all-available-devices 2022-03-21T14:55:16.469 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.472208 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-21T14:55:16.470 INFO:tasks.rook.operator.smithi043.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-21T14:55:16.470 INFO:tasks.rook.operator.smithi043.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-21T14:55:16.470 INFO:tasks.rook.operator.smithi043.stdout:   ... // 3 identical fields 2022-03-21T14:55:16.471 INFO:tasks.rook.operator.smithi043.stdout:   Config: nil, 2022-03-21T14:55:16.471 INFO:tasks.rook.operator.smithi043.stdout:   Selection: {}, 2022-03-21T14:55:16.471 INFO:tasks.rook.operator.smithi043.stdout:-  StorageClassDeviceSets: nil, 2022-03-21T14:55:16.471 INFO:tasks.rook.operator.smithi043.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-21T14:55:16.471 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.472 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi043-lv-1", 2022-03-21T14:55:16.472 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.472 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.472 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.473 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.473 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi043-lv-2", 2022-03-21T14:55:16.473 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.474 INFO:teuthology.orchestra.run.smithi043.stdout:['Success'] 2022-03-21T14:55:16.480 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.480 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.480 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.480 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi043-lv-3", 2022-03-21T14:55:16.480 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.481 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.481 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.481 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.481 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi043-lv-4", 2022-03-21T14:55:16.482 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.482 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.482 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.482 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.483 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi065-lv-1", 2022-03-21T14:55:16.483 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.483 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.483 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.484 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.484 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi065-lv-2", 2022-03-21T14:55:16.484 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.484 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.484 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.485 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.485 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi065-lv-3", 2022-03-21T14:55:16.485 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.485 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.486 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.486 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.486 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi065-lv-4", 2022-03-21T14:55:16.486 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.486 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.487 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.487 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.487 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi154-lv-1", 2022-03-21T14:55:16.487 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.488 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.488 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.488 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.488 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi154-lv-2", 2022-03-21T14:55:16.489 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.489 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.489 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.489 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.489 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi154-lv-3", 2022-03-21T14:55:16.490 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.490 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.490 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.490 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:55:16.491 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi154-lv-4", 2022-03-21T14:55:16.491 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:55:16.491 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-21T14:55:16.491 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.492 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:55:16.492 INFO:tasks.rook.operator.smithi043.stdout:   }, 2022-03-21T14:55:16.492 INFO:tasks.rook.operator.smithi043.stdout:   Annotations: nil, 2022-03-21T14:55:16.492 INFO:tasks.rook.operator.smithi043.stdout:   Labels: nil, 2022-03-21T14:55:16.492 INFO:tasks.rook.operator.smithi043.stdout:   ... // 20 identical fields 2022-03-21T14:55:16.493 INFO:tasks.rook.operator.smithi043.stdout:  } 2022-03-21T14:55:16.493 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.472350 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-21T14:55:16.493 INFO:tasks.rook.operator.smithi043.stdout:I0321 14:55:16.472648 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-21T14:55:16.493 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.472666 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-21T14:55:16.494 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.472765 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-21T14:55:16.494 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.472793 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-21T14:55:16.494 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.472823 I | operator: successfully started the controller-runtime manager 2022-03-21T14:55:16.495 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.493116 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-21T14:55:16.495 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.493152 I | operator: watching all namespaces for Ceph CRs 2022-03-21T14:55:16.495 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.493209 I | operator: setting up schemes 2022-03-21T14:55:16.496 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.497313 I | operator: setting up the controller-runtime manager 2022-03-21T14:55:16.497 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-21T14:55:16.497 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:55:16.698 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.702030 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-21T14:55:16.704 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.704952 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-21T14:55:16.704 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.704984 I | ceph-cluster-controller: successfully started 2022-03-21T14:55:16.705 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705041 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-21T14:55:16.705 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705075 I | ceph-crashcollector-controller: successfully started 2022-03-21T14:55:16.705 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705100 I | ceph-block-pool-controller: successfully started 2022-03-21T14:55:16.705 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705129 I | ceph-object-store-user-controller: successfully started 2022-03-21T14:55:16.705 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705158 I | ceph-object-realm-controller: successfully started 2022-03-21T14:55:16.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705177 I | ceph-object-zonegroup-controller: successfully started 2022-03-21T14:55:16.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705189 I | ceph-object-zone-controller: successfully started 2022-03-21T14:55:16.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705316 I | ceph-object-controller: successfully started 2022-03-21T14:55:16.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705359 I | ceph-file-controller: successfully started 2022-03-21T14:55:16.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705386 I | ceph-nfs-controller: successfully started 2022-03-21T14:55:16.707 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705422 I | ceph-rbd-mirror-controller: successfully started 2022-03-21T14:55:16.707 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705449 I | ceph-client-controller: successfully started 2022-03-21T14:55:16.707 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705469 I | ceph-filesystem-mirror-controller: successfully started 2022-03-21T14:55:16.707 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705501 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-21T14:55:16.707 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705518 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-21T14:55:16.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705531 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-21T14:55:16.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705546 I | ceph-bucket-topic: successfully started 2022-03-21T14:55:16.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705558 I | ceph-bucket-notification: successfully started 2022-03-21T14:55:16.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705574 I | ceph-bucket-notification: successfully started 2022-03-21T14:55:16.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.705590 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-21T14:55:16.709 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.706802 I | operator: starting the controller-runtime manager 2022-03-21T14:55:16.805 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.809095 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-21T14:55:16.806 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.809127 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-21T14:55:16.806 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.809136 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-21T14:55:16.815 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.813380 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:55:16.815 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.813863 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-21T14:55:16.815 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.817134 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:55:16.815 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.817202 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-21T14:55:16.816 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.817216 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-21T14:55:16.816 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.818357 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:55:16.818 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.819151 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-21T14:55:16.818 INFO:tasks.rook.operator.smithi043.stdout:I0321 14:55:16.819203 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-21T14:55:16.841 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.845641 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-21T14:55:16.864 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.868213 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-21T14:55:16.898 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.901854 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-21T14:55:16.898 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.901886 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-21T14:55:16.898 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.901899 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-21T14:55:16.899 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.901911 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-21T14:55:16.899 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.901923 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-21T14:55:16.900 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.901938 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-21T14:55:16.900 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.901955 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-21T14:55:16.900 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.901972 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-21T14:55:16.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.901989 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-21T14:55:16.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.902006 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-21T14:55:16.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.902021 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-21T14:55:16.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.902039 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-21T14:55:16.902 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.902055 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-21T14:55:16.902 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.902068 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-21T14:55:16.902 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.902082 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-21T14:55:16.902 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.902098 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-21T14:55:16.902 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.902188 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:55:16.903 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:16.902211 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:55:17.028 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:55:17.047 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-21T14:55:17.047 INFO:tasks.rook: have 0/12 OSDs 2022-03-21T14:55:23.052 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054828 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-21T14:55:23.053 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054867 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-21T14:55:23.053 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054877 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-21T14:55:23.053 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054882 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-21T14:55:23.054 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054889 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-21T14:55:23.054 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054894 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-21T14:55:23.054 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054899 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-21T14:55:23.054 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054904 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-21T14:55:23.055 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054909 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-21T14:55:23.055 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054915 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-21T14:55:23.057 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054923 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-21T14:55:23.057 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054929 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-21T14:55:23.058 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054935 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-21T14:55:23.058 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054944 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-21T14:55:23.058 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054951 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-21T14:55:23.058 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054960 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-21T14:55:23.059 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054966 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-21T14:55:23.059 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054975 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-21T14:55:23.059 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054982 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-21T14:55:23.060 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054991 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-21T14:55:23.060 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.054997 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-21T14:55:23.060 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.055002 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-21T14:55:23.060 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.055010 I | ceph-csi: Kubernetes version is 1.23 2022-03-21T14:55:23.060 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.055017 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-21T14:55:23.061 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.055026 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-21T14:55:23.310 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.314590 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-21T14:55:23.311 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.314620 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-21T14:55:23.315 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.315655 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-21T14:55:23.323 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.327505 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:55:23.324 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.327525 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:55:23.324 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.327529 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:55:23.324 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.327531 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:55:23.324 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.327534 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:55:23.325 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.327536 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:55:23.325 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.327539 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-21T14:55:23.486 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.490806 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:55:23.518 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.522726 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:55:23.523 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.523330 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:55:23.523 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.523344 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:55:23.523 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.523350 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-21T14:55:23.579 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.583707 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-21T14:55:23.678 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.683146 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:55:23.718 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.722774 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:55:23.719 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.722793 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:55:23.719 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.722797 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-21T14:55:23.838 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.841926 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:55:23.839 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.841961 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:55:23.840 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.841970 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-21T14:55:23.891 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:23.895139 I | ceph-csi: successfully started CSI CephFS driver 2022-03-21T14:55:24.071 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:24.075075 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-21T14:55:24.075 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:24.079482 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-21T14:55:24.076 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:24.079496 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-21T14:55:24.092 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:24.096642 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-21T14:55:24.208 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:24.212723 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-21T14:55:24.830 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:24.834008 I | op-mon: start running mons 2022-03-21T14:55:25.042 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:25.046426 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:55:25.162 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:25.166146 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789","10.111.185.43:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:55:25.761 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:25.765003 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:55:25.761 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:25.765233 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:55:27.049 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:55:27.361 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:27.365768 I | op-mon: targeting the mon count 3 2022-03-21T14:55:27.366 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:27.371310 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:55:27.583 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:55:27.607 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-21T14:55:27.608 INFO:tasks.rook: have 0/12 OSDs 2022-03-21T14:55:27.739 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:27.743487 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:55:27.740 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:27.743519 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:55:28.089 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:28.092796 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:55:28.089 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:28.092835 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:55:28.424 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:28.427905 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:55:28.424 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:28.427936 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:55:28.766 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:28.769807 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:55:28.766 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:28.769838 I | op-config: deleting "log file" option from the mon configuration database 2022-03-21T14:55:29.117 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:29.120762 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-21T14:55:29.117 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:29.120792 I | op-mon: checking for basic quorum with existing mons 2022-03-21T14:55:29.214 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:29.218644 I | op-mon: mon "b" ip is 10.96.127.40 2022-03-21T14:55:29.374 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:29.378089 I | op-mon: mon "c" ip is 10.111.185.43 2022-03-21T14:55:29.763 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:29.767872 I | op-mon: mon "a" ip is 10.97.100.133 2022-03-21T14:55:30.361 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:30.366055 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789","10.111.185.43:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:55:30.961 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:30.965567 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:55:30.962 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:30.965814 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:55:31.399 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:31.404181 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-21T14:55:31.414 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:31.418514 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-21T14:55:31.414 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:31.418547 I | op-mon: waiting for mon quorum with [b c a] 2022-03-21T14:55:31.995 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:31.999173 I | op-mon: mons running: [b c a] 2022-03-21T14:55:32.427 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:32.430972 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:55:32.436 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:32.441265 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-21T14:55:32.447 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:32.451507 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-21T14:55:32.447 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:32.451529 I | op-mon: waiting for mon quorum with [b c a] 2022-03-21T14:55:32.564 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:32.569069 I | op-mon: mons running: [b c a] 2022-03-21T14:55:32.973 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:32.977410 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:55:32.982 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:32.986552 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-21T14:55:32.991 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:32.996216 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-21T14:55:32.992 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:32.996243 I | op-mon: waiting for mon quorum with [b c a] 2022-03-21T14:55:33.167 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:33.171120 I | op-mon: mons running: [b c a] 2022-03-21T14:55:33.581 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:33.585458 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:55:33.581 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:33.585478 I | op-mon: mons created: 3 2022-03-21T14:55:34.036 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:34.040366 I | op-mon: waiting for mon quorum with [b c a] 2022-03-21T14:55:34.073 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:34.078127 I | op-mon: mons running: [b c a] 2022-03-21T14:55:34.481 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:34.485051 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:55:34.481 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:34.485095 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-21T14:55:34.910 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:34.914241 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-21T14:55:35.330 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:35.334119 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-21T14:55:35.740 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:35.744304 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-21T14:55:36.247 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:36.251492 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-21T14:55:36.247 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:36.251523 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-21T14:55:36.674 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:36.678438 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-21T14:55:36.674 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:36.678473 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-21T14:55:36.675 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:36.678483 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-21T14:55:37.096 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:37.100809 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-21T14:55:37.149 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:37.154113 I | op-mgr: start running mgr 2022-03-21T14:55:37.149 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:37.154149 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-21T14:55:37.610 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:55:37.691 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:37.695722 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-21T14:55:37.743 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:37.747922 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-21T14:55:37.747 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:37.751791 I | op-mgr: setting services to point to mgr "a" 2022-03-21T14:55:38.148 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:55:38.167 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-21T14:55:38.168 INFO:tasks.rook: have 0/12 OSDs 2022-03-21T14:55:38.227 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.231934 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-21T14:55:38.234 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.232069 I | op-mgr: successful modules: balancer 2022-03-21T14:55:38.244 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.249453 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-21T14:55:38.245 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.249486 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-21T14:55:38.260 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.264793 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-21T14:55:38.427 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.432185 I | op-osd: creating 1 new PVCs for device set "smithi043-lv-1" 2022-03-21T14:55:38.629 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.634303 I | op-osd: successfully provisioned PVC "smithi043-lv-1-data-0vfptn" 2022-03-21T14:55:38.630 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.634324 I | op-osd: creating 1 new PVCs for device set "smithi043-lv-2" 2022-03-21T14:55:38.830 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.834834 I | op-osd: successfully provisioned PVC "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:55:38.831 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:38.834876 I | op-osd: creating 1 new PVCs for device set "smithi043-lv-3" 2022-03-21T14:55:39.029 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.033947 I | op-osd: successfully provisioned PVC "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:55:39.030 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.033984 I | op-osd: creating 1 new PVCs for device set "smithi043-lv-4" 2022-03-21T14:55:39.230 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.234783 I | op-osd: successfully provisioned PVC "smithi043-lv-4-data-0zg56f" 2022-03-21T14:55:39.264 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.234823 I | op-osd: creating 1 new PVCs for device set "smithi065-lv-1" 2022-03-21T14:55:39.429 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.433692 I | op-osd: successfully provisioned PVC "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:55:39.430 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.433725 I | op-osd: creating 1 new PVCs for device set "smithi065-lv-2" 2022-03-21T14:55:39.627 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.631755 I | op-osd: successfully provisioned PVC "smithi065-lv-2-data-0fgs82" 2022-03-21T14:55:39.628 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.631795 I | op-osd: creating 1 new PVCs for device set "smithi065-lv-3" 2022-03-21T14:55:39.760 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.764855 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-21T14:55:39.761 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.764937 I | op-mgr: successful modules: dashboard 2022-03-21T14:55:39.812 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.816803 I | op-mgr: successful modules: prometheus 2022-03-21T14:55:39.827 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.831928 I | op-osd: successfully provisioned PVC "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:55:39.827 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:39.831959 I | op-osd: creating 1 new PVCs for device set "smithi065-lv-4" 2022-03-21T14:55:40.050 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.054509 I | op-osd: successfully provisioned PVC "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:55:40.050 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.054544 I | op-osd: creating 1 new PVCs for device set "smithi154-lv-1" 2022-03-21T14:55:40.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.233956 I | op-osd: successfully provisioned PVC "smithi154-lv-1-data-058jxx" 2022-03-21T14:55:40.230 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.233995 I | op-osd: creating 1 new PVCs for device set "smithi154-lv-2" 2022-03-21T14:55:40.429 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.434209 I | op-osd: successfully provisioned PVC "smithi154-lv-2-data-0n64sd" 2022-03-21T14:55:40.430 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.434238 I | op-osd: creating 1 new PVCs for device set "smithi154-lv-3" 2022-03-21T14:55:40.627 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.631974 I | op-osd: successfully provisioned PVC "smithi154-lv-3-data-0vf74c" 2022-03-21T14:55:40.628 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.632009 I | op-osd: creating 1 new PVCs for device set "smithi154-lv-4" 2022-03-21T14:55:40.828 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.832839 I | op-osd: successfully provisioned PVC "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:55:40.893 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:40.898303 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-1-data-0vfptn" 2022-03-21T14:55:41.257 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:41.262169 I | op-osd: started OSD provisioning job for PVC "smithi043-lv-1-data-0vfptn" 2022-03-21T14:55:41.258 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:41.262209 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:55:41.625 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:41.629670 I | op-mgr: successful modules: orchestrator modules 2022-03-21T14:55:41.671 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:41.675570 I | op-osd: started OSD provisioning job for PVC "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:55:41.671 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:41.675606 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:55:42.057 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:42.061472 I | op-osd: started OSD provisioning job for PVC "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:55:42.057 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:42.061512 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-4-data-0zg56f" 2022-03-21T14:55:43.091 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:43.095568 I | op-osd: started OSD provisioning job for PVC "smithi043-lv-4-data-0zg56f" 2022-03-21T14:55:43.091 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:43.095607 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:55:43.292 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:43.295736 I | op-osd: started OSD provisioning job for PVC "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:55:43.292 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:43.295770 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-2-data-0fgs82" 2022-03-21T14:55:43.794 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:43.799081 I | op-osd: started OSD provisioning job for PVC "smithi065-lv-2-data-0fgs82" 2022-03-21T14:55:43.795 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:43.799118 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:55:44.431 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:44.435869 I | op-osd: started OSD provisioning job for PVC "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:55:44.431 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:44.435913 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:55:44.850 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:44.854552 I | op-osd: started OSD provisioning job for PVC "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:55:44.850 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:44.854587 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-1-data-058jxx" 2022-03-21T14:55:45.159 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:45.163479 I | op-osd: started OSD provisioning job for PVC "smithi154-lv-1-data-058jxx" 2022-03-21T14:55:45.159 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:45.163507 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-2-data-0n64sd" 2022-03-21T14:55:45.485 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:45.490228 I | op-osd: started OSD provisioning job for PVC "smithi154-lv-2-data-0n64sd" 2022-03-21T14:55:45.486 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:45.490267 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-3-data-0vf74c" 2022-03-21T14:55:46.193 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:46.197017 I | op-osd: started OSD provisioning job for PVC "smithi154-lv-3-data-0vf74c" 2022-03-21T14:55:46.193 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:46.197050 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:55:46.987 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:46.992119 I | op-osd: started OSD provisioning job for PVC "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:55:46.988 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:46.992147 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-21T14:55:46.988 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:46.992151 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-21T14:55:47.208 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.211877 I | op-osd: OSD orchestration status for PVC smithi043-lv-1-data-0vfptn is "starting" 2022-03-21T14:55:47.208 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.211926 I | op-osd: OSD orchestration status for PVC smithi043-lv-2-data-0jwsgt is "starting" 2022-03-21T14:55:47.208 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.211962 I | op-osd: OSD orchestration status for PVC smithi043-lv-3-data-0p8lrq is "starting" 2022-03-21T14:55:47.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.212013 I | op-osd: OSD orchestration status for PVC smithi043-lv-4-data-0zg56f is "starting" 2022-03-21T14:55:47.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.212065 I | op-osd: OSD orchestration status for PVC smithi065-lv-1-data-0zrnzn is "starting" 2022-03-21T14:55:47.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.212126 I | op-osd: OSD orchestration status for PVC smithi065-lv-2-data-0fgs82 is "starting" 2022-03-21T14:55:47.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.212172 I | op-osd: OSD orchestration status for PVC smithi065-lv-3-data-0pnlwn is "starting" 2022-03-21T14:55:47.210 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.212210 I | op-osd: OSD orchestration status for PVC smithi065-lv-4-data-0jcxd2 is "starting" 2022-03-21T14:55:47.210 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.212248 I | op-osd: OSD orchestration status for PVC smithi154-lv-1-data-058jxx is "starting" 2022-03-21T14:55:47.210 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.212288 I | op-osd: OSD orchestration status for PVC smithi154-lv-2-data-0n64sd is "starting" 2022-03-21T14:55:47.210 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.212318 I | op-osd: OSD orchestration status for PVC smithi154-lv-3-data-0vf74c is "starting" 2022-03-21T14:55:47.210 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:47.212360 I | op-osd: OSD orchestration status for PVC smithi154-lv-4-data-0h7xg6 is "starting" 2022-03-21T14:55:48.169 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:55:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:55:49.393 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-21T14:55:49.394 INFO:tasks.rook: have 0/12 OSDs 2022-03-21T14:55:53.953 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:53.958187 I | op-osd: OSD orchestration status for node smithi154-lv-1-data-058jxx is "orchestrating" 2022-03-21T14:55:54.157 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:54.162232 I | op-osd: OSD orchestration status for node smithi065-lv-2-data-0fgs82 is "orchestrating" 2022-03-21T14:55:54.157 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:54.162319 I | op-osd: OSD orchestration status for node smithi065-lv-3-data-0pnlwn is "orchestrating" 2022-03-21T14:55:54.509 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:54.514131 I | op-osd: OSD orchestration status for PVC smithi154-lv-1-data-058jxx is "orchestrating" 2022-03-21T14:55:54.517 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:54.522195 I | op-osd: OSD orchestration status for PVC smithi065-lv-3-data-0pnlwn is "orchestrating" 2022-03-21T14:55:54.549 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:54.553808 I | op-osd: OSD orchestration status for PVC smithi065-lv-2-data-0fgs82 is "orchestrating" 2022-03-21T14:55:57.509 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:57.513594 I | op-osd: OSD orchestration status for node smithi043-lv-2-data-0jwsgt is "orchestrating" 2022-03-21T14:55:57.509 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:57.513702 I | op-osd: OSD orchestration status for node smithi043-lv-3-data-0p8lrq is "orchestrating" 2022-03-21T14:55:58.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:58.234558 I | op-osd: OSD orchestration status for PVC smithi043-lv-2-data-0jwsgt is "orchestrating" 2022-03-21T14:55:58.480 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:55:58.484738 I | op-osd: OSD orchestration status for PVC smithi043-lv-3-data-0p8lrq is "orchestrating" 2022-03-21T14:55:59.395 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:56:00.494 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:56:00.519 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":5,"num_osds":3,"num_up_osds":0,"osd_up_since":0,"num_in_osds":3,"osd_in_since":1647874557,"num_remapped_pgs":0} 2022-03-21T14:56:00.520 INFO:tasks.rook: have 0/12 OSDs 2022-03-21T14:56:01.197 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:01.202335 I | op-osd: OSD orchestration status for PVC smithi154-lv-1-data-058jxx is "completed" 2022-03-21T14:56:01.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:01.202358 I | op-osd: creating OSD 0 on PVC "smithi154-lv-1-data-058jxx" 2022-03-21T14:56:01.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:01.202368 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-1-data-058jxx" 2022-03-21T14:56:03.699 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:03.702763 I | op-osd: OSD orchestration status for PVC smithi065-lv-3-data-0pnlwn is "completed" 2022-03-21T14:56:03.700 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:03.702790 I | op-osd: creating OSD 1 on PVC "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:56:03.700 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:03.702801 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:56:04.778 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:04.782686 I | op-osd: OSD orchestration status for PVC smithi065-lv-2-data-0fgs82 is "completed" 2022-03-21T14:56:04.779 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:04.782721 I | op-osd: creating OSD 2 on PVC "smithi065-lv-2-data-0fgs82" 2022-03-21T14:56:04.779 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:04.782736 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-2-data-0fgs82" 2022-03-21T14:56:06.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:06.560785 I | op-osd: OSD orchestration status for PVC smithi043-lv-3-data-0p8lrq is "completed" 2022-03-21T14:56:06.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:06.560821 I | op-osd: creating OSD 3 on PVC "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:56:06.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:06.560834 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:56:08.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:08.213849 I | op-osd: OSD orchestration status for PVC smithi043-lv-2-data-0jwsgt is "completed" 2022-03-21T14:56:08.210 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:08.213914 I | op-osd: creating OSD 4 on PVC "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:56:08.211 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:08.213935 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:56:08.611 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:08.612805 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi043": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T14:56:08.651 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:08.655146 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi043": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T14:56:08.937 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:08.803754 I | op-osd: OSD orchestration status for node smithi065-lv-4-data-0jcxd2 is "orchestrating" 2022-03-21T14:56:08.938 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:08.803869 I | op-osd: OSD orchestration status for node smithi065-lv-1-data-0zrnzn is "orchestrating" 2022-03-21T14:56:08.938 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:08.851184 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi043": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T14:56:09.131 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:09.135957 I | op-osd: OSD orchestration status for PVC smithi065-lv-1-data-0zrnzn is "orchestrating" 2022-03-21T14:56:10.159 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:10.164034 I | op-osd: OSD orchestration status for PVC smithi065-lv-4-data-0jcxd2 is "orchestrating" 2022-03-21T14:56:10.521 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:56:11.410 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:11.414648 I | op-osd: OSD orchestration status for node smithi154-lv-3-data-0vf74c is "orchestrating" 2022-03-21T14:56:12.509 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:12.514646 I | op-osd: OSD orchestration status for PVC smithi154-lv-3-data-0vf74c is "orchestrating" 2022-03-21T14:56:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:56:12.710 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"num_osds":6,"num_up_osds":0,"osd_up_since":0,"num_in_osds":6,"osd_in_since":1647874571,"num_remapped_pgs":0} 2022-03-21T14:56:12.711 INFO:tasks.rook: have 0/12 OSDs 2022-03-21T14:56:13.002 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:13.006961 I | op-osd: OSD orchestration status for node smithi154-lv-2-data-0n64sd is "orchestrating" 2022-03-21T14:56:13.047 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:13.052242 I | op-osd: OSD orchestration status for node smithi154-lv-4-data-0h7xg6 is "orchestrating" 2022-03-21T14:56:13.325 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:13.330654 I | op-osd: OSD orchestration status for node smithi043-lv-4-data-0zg56f is "orchestrating" 2022-03-21T14:56:13.352 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:13.357830 I | op-osd: OSD orchestration status for node smithi043-lv-1-data-0vfptn is "orchestrating" 2022-03-21T14:56:13.691 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:13.696259 I | op-osd: OSD orchestration status for PVC smithi154-lv-4-data-0h7xg6 is "orchestrating" 2022-03-21T14:56:13.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:13.733160 I | op-osd: OSD orchestration status for PVC smithi154-lv-2-data-0n64sd is "orchestrating" 2022-03-21T14:56:13.747 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:13.752665 I | op-osd: OSD orchestration status for PVC smithi043-lv-4-data-0zg56f is "orchestrating" 2022-03-21T14:56:13.860 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:13.866158 I | op-osd: OSD orchestration status for PVC smithi043-lv-1-data-0vfptn is "orchestrating" 2022-03-21T14:56:17.369 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:17.373906 I | op-osd: OSD orchestration status for PVC smithi065-lv-1-data-0zrnzn is "completed" 2022-03-21T14:56:17.369 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:17.373960 I | op-osd: creating OSD 5 on PVC "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:56:17.369 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:17.373986 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:56:20.368 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:20.373199 I | op-osd: OSD orchestration status for PVC smithi065-lv-4-data-0jcxd2 is "completed" 2022-03-21T14:56:20.369 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:20.373222 I | op-osd: creating OSD 6 on PVC "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:56:20.369 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:20.373236 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:56:21.353 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:21.358251 I | op-osd: OSD orchestration status for PVC smithi154-lv-3-data-0vf74c is "completed" 2022-03-21T14:56:21.354 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:21.358288 I | op-osd: creating OSD 7 on PVC "smithi154-lv-3-data-0vf74c" 2022-03-21T14:56:21.354 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:21.358307 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-3-data-0vf74c" 2022-03-21T14:56:22.474 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:22.479823 I | op-osd: OSD orchestration status for PVC smithi154-lv-4-data-0h7xg6 is "completed" 2022-03-21T14:56:22.475 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:22.479857 I | op-osd: creating OSD 9 on PVC "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:56:22.475 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:22.479880 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:56:22.712 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:56:23.464 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:23.469305 I | op-osd: OSD orchestration status for PVC smithi154-lv-2-data-0n64sd is "completed" 2022-03-21T14:56:23.464 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:23.469356 I | op-osd: creating OSD 8 on PVC "smithi154-lv-2-data-0n64sd" 2022-03-21T14:56:23.464 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:23.469381 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-2-data-0n64sd" 2022-03-21T14:56:25.217 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:56:25.227 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:25.232977 I | op-osd: OSD orchestration status for PVC smithi043-lv-4-data-0zg56f is "completed" 2022-03-21T14:56:25.228 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:25.233008 I | op-osd: creating OSD 10 on PVC "smithi043-lv-4-data-0zg56f" 2022-03-21T14:56:25.228 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:25.233019 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-4-data-0zg56f" 2022-03-21T14:56:25.237 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1647874577,"num_remapped_pgs":0} 2022-03-21T14:56:25.237 INFO:tasks.rook: have 0/12 OSDs 2022-03-21T14:56:26.451 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:26.456756 I | op-osd: OSD orchestration status for PVC smithi043-lv-1-data-0vfptn is "completed" 2022-03-21T14:56:26.451 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:26.456795 I | op-osd: creating OSD 11 on PVC "smithi043-lv-1-data-0vfptn" 2022-03-21T14:56:26.452 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:26.456810 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-1-data-0vfptn" 2022-03-21T14:56:29.811 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:29.816974 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-21T14:56:29.812 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:56:29.817000 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:56:35.239 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:56:36.611 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:56:36.669 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":19,"num_osds":12,"num_up_osds":2,"osd_up_since":1647874593,"num_in_osds":12,"osd_in_since":1647874577,"num_remapped_pgs":0} 2022-03-21T14:56:36.669 INFO:tasks.rook: have 2/12 OSDs 2022-03-21T14:56:46.670 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:56:47.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:56:47.275 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":25,"num_osds":12,"num_up_osds":4,"osd_up_since":1647874603,"num_in_osds":12,"osd_in_since":1647874577,"num_remapped_pgs":1} 2022-03-21T14:56:47.276 INFO:tasks.rook: have 4/12 OSDs 2022-03-21T14:56:57.277 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:56:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:56:57.837 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":30,"num_osds":12,"num_up_osds":10,"osd_up_since":1647874611,"num_in_osds":12,"osd_in_since":1647874577,"num_remapped_pgs":0} 2022-03-21T14:56:57.837 INFO:tasks.rook: have 10/12 OSDs 2022-03-21T14:57:07.838 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd stat -f json 2022-03-21T14:57:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:08.413 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":37,"num_osds":12,"num_up_osds":12,"osd_up_since":1647874620,"num_in_osds":12,"osd_in_since":1647874577,"num_remapped_pgs":0} 2022-03-21T14:57:08.413 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-21T14:57:08.414 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- cat /etc/ceph/ceph.conf 2022-03-21T14:57:08.570 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2022-03-21T14:57:08.571 INFO:teuthology.orchestra.run.smithi043.stdout:mon_host = 10.97.100.133:6789,10.96.127.40:6789,10.111.185.43:6789 2022-03-21T14:57:08.571 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:08.571 INFO:teuthology.orchestra.run.smithi043.stdout:[client.admin] 2022-03-21T14:57:08.571 INFO:teuthology.orchestra.run.smithi043.stdout:keyring = /etc/ceph/keyring 2022-03-21T14:57:08.575 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- cat /etc/ceph/keyring 2022-03-21T14:57:08.780 INFO:teuthology.orchestra.run.smithi043.stdout:[client.admin] 2022-03-21T14:57:08.780 INFO:teuthology.orchestra.run.smithi043.stdout:key = AQAbkThioH8VHhAAfCKSPLRtqcjC8ofCIJpH3w== 2022-03-21T14:57:08.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph 2022-03-21T14:57:08.801 DEBUG:teuthology.orchestra.run.smithi065:> sudo mkdir -p /etc/ceph 2022-03-21T14:57:08.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2022-03-21T14:57:08.827 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:57:08.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.conf 2022-03-21T14:57:08.884 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:57:08.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/keyring 2022-03-21T14:57:08.899 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:57:08.900 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd of=/etc/ceph/ceph.conf 2022-03-21T14:57:08.914 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:57:08.915 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd of=/etc/ceph/keyring 2022-03-21T14:57:08.928 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:57:08.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.conf 2022-03-21T14:57:08.944 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:57:08.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/keyring 2022-03-21T14:57:09.003 INFO:tasks.rook:Setting up client nodes... 2022-03-21T14:57:09.004 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-21T14:57:09.762 INFO:teuthology.orchestra.run.smithi043.stdout:[client.a] 2022-03-21T14:57:09.763 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBFkjhi0q0VJhAAkcnCmm0mrgB9qT0KdidC4A== 2022-03-21T14:57:09.787 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-03-21T14:57:09.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-21T14:57:09.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-21T14:57:09.811 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-21T14:57:10.555 INFO:teuthology.orchestra.run.smithi043.stdout:[client.b] 2022-03-21T14:57:10.556 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBGkjhiHlgfGxAAa9iHqMadgkEHDvtJ3OfGfA== 2022-03-21T14:57:10.584 DEBUG:teuthology.orchestra.run.smithi065:> set -ex 2022-03-21T14:57:10.585 DEBUG:teuthology.orchestra.run.smithi065:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-21T14:57:10.585 DEBUG:teuthology.orchestra.run.smithi065:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-21T14:57:10.608 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-21T14:57:11.332 INFO:teuthology.orchestra.run.smithi043.stdout:[client.c] 2022-03-21T14:57:11.332 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBHkjhiIlOgDhAAmNjt6UZ03trSzA++PJlgMg== 2022-03-21T14:57:11.355 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2022-03-21T14:57:11.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-21T14:57:11.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-21T14:57:11.380 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd dump --format=json 2022-03-21T14:57:11.907 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:11.907 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":37,"fsid":"39ceab48-33b1-458f-a341-29eb1474d30e","created":"2022-03-21T14:52:35.072339+0000","modified":"2022-03-21T14:57:05.094238+0000","last_up_change":"2022-03-21T14:57:00.443473+0000","last_in_change":"2022-03-21T14:56:17.751473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:56:40.229199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"db68da4b-594f-42ab-90bd-8891ce275c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6800","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6801","nonce":77724832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6802","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6803","nonce":77724832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6806","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6807","nonce":77724832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6804","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6805","nonce":77724832}]},"public_addr":"10.249.82.13:6801/77724832","cluster_addr":"10.249.82.13:6803/77724832","heartbeat_back_addr":"10.249.82.13:6807/77724832","heartbeat_front_addr":"10.249.82.13:6805/77724832","state":["exists","up"]},{"osd":1,"uuid":"d5614f90-f43f-4424-a246-23347f5c3bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6800","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6801","nonce":424166575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6802","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6803","nonce":424166575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6806","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6807","nonce":424166575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6804","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6805","nonce":424166575}]},"public_addr":"10.249.81.15:6801/424166575","cluster_addr":"10.249.81.15:6803/424166575","heartbeat_back_addr":"10.249.81.15:6807/424166575","heartbeat_front_addr":"10.249.81.15:6805/424166575","state":["exists","up"]},{"osd":2,"uuid":"1a7f1675-ec12-4c4f-af01-7e3a224c4265","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6800","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6801","nonce":550832532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6802","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6803","nonce":550832532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6806","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6807","nonce":550832532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6804","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6805","nonce":550832532}]},"public_addr":"10.249.81.16:6801/550832532","cluster_addr":"10.249.81.16:6803/550832532","heartbeat_back_addr":"10.249.81.16:6807/550832532","heartbeat_front_addr":"10.249.81.16:6805/550832532","state":["exists","up"]},{"osd":3,"uuid":"5a57fa32-a917-4784-a91e-2921e7218b40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6800","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6801","nonce":3508099279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6802","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6803","nonce":3508099279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6806","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6807","nonce":3508099279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6804","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6805","nonce":3508099279}]},"public_addr":"10.249.80.12:6801/3508099279","cluster_addr":"10.249.80.12:6803/3508099279","heartbeat_back_addr":"10.249.80.12:6807/3508099279","heartbeat_front_addr":"10.249.80.12:6805/3508099279","state":["exists","up"]},{"osd":4,"uuid":"0a26bc99-8855-4473-be5a-78b35f554d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6800","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6801","nonce":3874051968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6802","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6803","nonce":3874051968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6806","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6807","nonce":3874051968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6804","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6805","nonce":3874051968}]},"public_addr":"10.249.80.13:6801/3874051968","cluster_addr":"10.249.80.13:6803/3874051968","heartbeat_back_addr":"10.249.80.13:6807/3874051968","heartbeat_front_addr":"10.249.80.13:6805/3874051968","state":["exists","up"]},{"osd":5,"uuid":"1e82124e-368c-4fc5-9edc-bbeb2d81ff9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6800","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6801","nonce":2272372010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6802","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6803","nonce":2272372010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6806","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6807","nonce":2272372010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6804","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6805","nonce":2272372010}]},"public_addr":"10.249.81.17:6801/2272372010","cluster_addr":"10.249.81.17:6803/2272372010","heartbeat_back_addr":"10.249.81.17:6807/2272372010","heartbeat_front_addr":"10.249.81.17:6805/2272372010","state":["exists","up"]},{"osd":6,"uuid":"525ee236-4220-4194-824c-777f2b3cc5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6800","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6801","nonce":4147345364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6802","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6803","nonce":4147345364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6806","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6807","nonce":4147345364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6804","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6805","nonce":4147345364}]},"public_addr":"10.249.81.18:6801/4147345364","cluster_addr":"10.249.81.18:6803/4147345364","heartbeat_back_addr":"10.249.81.18:6807/4147345364","heartbeat_front_addr":"10.249.81.18:6805/4147345364","state":["exists","up"]},{"osd":7,"uuid":"be39cf83-817f-4c1c-9c32-b1edad4a985c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6800","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6801","nonce":4147162602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6802","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6803","nonce":4147162602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6806","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6807","nonce":4147162602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6804","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6805","nonce":4147162602}]},"public_addr":"10.249.82.14:6801/4147162602","cluster_addr":"10.249.82.14:6803/4147162602","heartbeat_back_addr":"10.249.82.14:6807/4147162602","heartbeat_front_addr":"10.249.82.14:6805/4147162602","state":["exists","up"]},{"osd":8,"uuid":"2b14b01b-a3e2-4e55-9532-768d8b53378d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6800","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6801","nonce":3981665281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6802","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6803","nonce":3981665281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6806","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6807","nonce":3981665281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6804","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6805","nonce":3981665281}]},"public_addr":"10.249.82.16:6801/3981665281","cluster_addr":"10.249.82.16:6803/3981665281","heartbeat_back_addr":"10.249.82.16:6807/3981665281","heartbeat_front_addr":"10.249.82.16:6805/3981665281","state":["exists","up"]},{"osd":9,"uuid":"7494a9de-3a85-43c3-be44-e5ead46781ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6800","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6801","nonce":433732131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6802","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6803","nonce":433732131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6806","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6807","nonce":433732131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6804","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6805","nonce":433732131}]},"public_addr":"10.249.82.15:6801/433732131","cluster_addr":"10.249.82.15:6803/433732131","heartbeat_back_addr":"10.249.82.15:6807/433732131","heartbeat_front_addr":"10.249.82.15:6805/433732131","state":["exists","up"]},{"osd":10,"uuid":"e87e0787-5205-4211-b83a-263628693ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6800","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6801","nonce":3117418671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6802","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6803","nonce":3117418671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6806","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6807","nonce":3117418671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6804","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6805","nonce":3117418671}]},"public_addr":"10.249.80.15:6801/3117418671","cluster_addr":"10.249.80.15:6803/3117418671","heartbeat_back_addr":"10.249.80.15:6807/3117418671","heartbeat_front_addr":"10.249.80.15:6805/3117418671","state":["exists","up"]},{"osd":11,"uuid":"79a411c2-f1c8-42c2-8484-607eedabfba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6800","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6801","nonce":1997927166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6802","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6803","nonce":1997927166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6806","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6807","nonce":1997927166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6804","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6805","nonce":1997927166}]},"public_addr":"10.249.80.16:6801/1997927166","cluster_addr":"10.249.80.16:6803/1997927166","heartbeat_back_addr":"10.249.80.16:6807/1997927166","heartbeat_front_addr":"10.249.80.16:6805/1997927166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:25.126118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:26.315593+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:30.036390+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:45.248011+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:43.033473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:40.469337+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:42.671520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:48.203811+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:48.299728+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:49.486769+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:59.499523+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:59.399609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.80.6:0/4223835587":"2022-03-22T14:55:13.010685+0000","10.249.80.6:6800/724391104":"2022-03-22T14:55:13.010685+0000","10.249.80.6:6801/724391104":"2022-03-22T14:55:13.010685+0000","10.249.80.6:0/1128621226":"2022-03-22T14:55:13.010685+0000","10.249.80.6:0/3353249299":"2022-03-22T14:55:13.010685+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:57:11.926 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-21T14:56:40.229199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-21T14:57:11.927 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd pool get .mgr pg_num 2022-03-21T14:57:12.475 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2022-03-21T14:57:12.495 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-21T14:57:12.496 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-21T14:57:12.497 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph mgr dump --format=json 2022-03-21T14:57:13.108 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:13.128 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":12,"active_gid":24167,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.6:6800","nonce":2408534242},{"type":"v1","addr":"10.249.80.6:6801","nonce":2408534242}]},"active_addr":"10.249.80.6:6801/2408534242","active_change":"2022-03-21T14:55:13.044256+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.249.80.6:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.249.80.6:0","nonce":837844096}]},{"addrvec":[{"type":"v2","addr":"10.249.80.6:0","nonce":182054784}]},{"addrvec":[{"type":"v2","addr":"10.249.80.6:0","nonce":2226840757}]},{"addrvec":[{"type":"v2","addr":"10.249.80.6:0","nonce":97403203}]}]}} 2022-03-21T14:57:13.131 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-21T14:57:13.131 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-21T14:57:13.132 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd dump --format=json 2022-03-21T14:57:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:13.708 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":37,"fsid":"39ceab48-33b1-458f-a341-29eb1474d30e","created":"2022-03-21T14:52:35.072339+0000","modified":"2022-03-21T14:57:05.094238+0000","last_up_change":"2022-03-21T14:57:00.443473+0000","last_in_change":"2022-03-21T14:56:17.751473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:56:40.229199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"db68da4b-594f-42ab-90bd-8891ce275c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6800","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6801","nonce":77724832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6802","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6803","nonce":77724832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6806","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6807","nonce":77724832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6804","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6805","nonce":77724832}]},"public_addr":"10.249.82.13:6801/77724832","cluster_addr":"10.249.82.13:6803/77724832","heartbeat_back_addr":"10.249.82.13:6807/77724832","heartbeat_front_addr":"10.249.82.13:6805/77724832","state":["exists","up"]},{"osd":1,"uuid":"d5614f90-f43f-4424-a246-23347f5c3bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6800","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6801","nonce":424166575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6802","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6803","nonce":424166575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6806","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6807","nonce":424166575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6804","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6805","nonce":424166575}]},"public_addr":"10.249.81.15:6801/424166575","cluster_addr":"10.249.81.15:6803/424166575","heartbeat_back_addr":"10.249.81.15:6807/424166575","heartbeat_front_addr":"10.249.81.15:6805/424166575","state":["exists","up"]},{"osd":2,"uuid":"1a7f1675-ec12-4c4f-af01-7e3a224c4265","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6800","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6801","nonce":550832532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6802","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6803","nonce":550832532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6806","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6807","nonce":550832532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6804","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6805","nonce":550832532}]},"public_addr":"10.249.81.16:6801/550832532","cluster_addr":"10.249.81.16:6803/550832532","heartbeat_back_addr":"10.249.81.16:6807/550832532","heartbeat_front_addr":"10.249.81.16:6805/550832532","state":["exists","up"]},{"osd":3,"uuid":"5a57fa32-a917-4784-a91e-2921e7218b40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6800","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6801","nonce":3508099279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6802","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6803","nonce":3508099279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6806","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6807","nonce":3508099279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6804","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6805","nonce":3508099279}]},"public_addr":"10.249.80.12:6801/3508099279","cluster_addr":"10.249.80.12:6803/3508099279","heartbeat_back_addr":"10.249.80.12:6807/3508099279","heartbeat_front_addr":"10.249.80.12:6805/3508099279","state":["exists","up"]},{"osd":4,"uuid":"0a26bc99-8855-4473-be5a-78b35f554d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6800","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6801","nonce":3874051968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6802","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6803","nonce":3874051968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6806","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6807","nonce":3874051968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6804","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6805","nonce":3874051968}]},"public_addr":"10.249.80.13:6801/3874051968","cluster_addr":"10.249.80.13:6803/3874051968","heartbeat_back_addr":"10.249.80.13:6807/3874051968","heartbeat_front_addr":"10.249.80.13:6805/3874051968","state":["exists","up"]},{"osd":5,"uuid":"1e82124e-368c-4fc5-9edc-bbeb2d81ff9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6800","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6801","nonce":2272372010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6802","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6803","nonce":2272372010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6806","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6807","nonce":2272372010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6804","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6805","nonce":2272372010}]},"public_addr":"10.249.81.17:6801/2272372010","cluster_addr":"10.249.81.17:6803/2272372010","heartbeat_back_addr":"10.249.81.17:6807/2272372010","heartbeat_front_addr":"10.249.81.17:6805/2272372010","state":["exists","up"]},{"osd":6,"uuid":"525ee236-4220-4194-824c-777f2b3cc5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6800","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6801","nonce":4147345364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6802","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6803","nonce":4147345364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6806","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6807","nonce":4147345364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6804","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6805","nonce":4147345364}]},"public_addr":"10.249.81.18:6801/4147345364","cluster_addr":"10.249.81.18:6803/4147345364","heartbeat_back_addr":"10.249.81.18:6807/4147345364","heartbeat_front_addr":"10.249.81.18:6805/4147345364","state":["exists","up"]},{"osd":7,"uuid":"be39cf83-817f-4c1c-9c32-b1edad4a985c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6800","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6801","nonce":4147162602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6802","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6803","nonce":4147162602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6806","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6807","nonce":4147162602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6804","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6805","nonce":4147162602}]},"public_addr":"10.249.82.14:6801/4147162602","cluster_addr":"10.249.82.14:6803/4147162602","heartbeat_back_addr":"10.249.82.14:6807/4147162602","heartbeat_front_addr":"10.249.82.14:6805/4147162602","state":["exists","up"]},{"osd":8,"uuid":"2b14b01b-a3e2-4e55-9532-768d8b53378d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6800","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6801","nonce":3981665281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6802","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6803","nonce":3981665281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6806","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6807","nonce":3981665281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6804","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6805","nonce":3981665281}]},"public_addr":"10.249.82.16:6801/3981665281","cluster_addr":"10.249.82.16:6803/3981665281","heartbeat_back_addr":"10.249.82.16:6807/3981665281","heartbeat_front_addr":"10.249.82.16:6805/3981665281","state":["exists","up"]},{"osd":9,"uuid":"7494a9de-3a85-43c3-be44-e5ead46781ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6800","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6801","nonce":433732131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6802","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6803","nonce":433732131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6806","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6807","nonce":433732131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6804","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6805","nonce":433732131}]},"public_addr":"10.249.82.15:6801/433732131","cluster_addr":"10.249.82.15:6803/433732131","heartbeat_back_addr":"10.249.82.15:6807/433732131","heartbeat_front_addr":"10.249.82.15:6805/433732131","state":["exists","up"]},{"osd":10,"uuid":"e87e0787-5205-4211-b83a-263628693ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6800","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6801","nonce":3117418671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6802","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6803","nonce":3117418671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6806","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6807","nonce":3117418671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6804","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6805","nonce":3117418671}]},"public_addr":"10.249.80.15:6801/3117418671","cluster_addr":"10.249.80.15:6803/3117418671","heartbeat_back_addr":"10.249.80.15:6807/3117418671","heartbeat_front_addr":"10.249.80.15:6805/3117418671","state":["exists","up"]},{"osd":11,"uuid":"79a411c2-f1c8-42c2-8484-607eedabfba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6800","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6801","nonce":1997927166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6802","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6803","nonce":1997927166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6806","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6807","nonce":1997927166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6804","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6805","nonce":1997927166}]},"public_addr":"10.249.80.16:6801/1997927166","cluster_addr":"10.249.80.16:6803/1997927166","heartbeat_back_addr":"10.249.80.16:6807/1997927166","heartbeat_front_addr":"10.249.80.16:6805/1997927166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:25.126118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:26.315593+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:30.036390+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:45.248011+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:43.033473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:40.469337+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:42.671520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:48.203811+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:48.299728+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:49.486769+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:59.499523+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:59.399609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.80.6:0/4223835587":"2022-03-22T14:55:13.010685+0000","10.249.80.6:6800/724391104":"2022-03-22T14:55:13.010685+0000","10.249.80.6:6801/724391104":"2022-03-22T14:55:13.010685+0000","10.249.80.6:0/1128621226":"2022-03-22T14:55:13.010685+0000","10.249.80.6:0/3353249299":"2022-03-22T14:55:13.010685+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:57:13.729 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-21T14:57:13.730 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd dump --format=json 2022-03-21T14:57:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:14.244 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":37,"fsid":"39ceab48-33b1-458f-a341-29eb1474d30e","created":"2022-03-21T14:52:35.072339+0000","modified":"2022-03-21T14:57:05.094238+0000","last_up_change":"2022-03-21T14:57:00.443473+0000","last_in_change":"2022-03-21T14:56:17.751473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-21T14:56:40.229199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"db68da4b-594f-42ab-90bd-8891ce275c7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6800","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6801","nonce":77724832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6802","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6803","nonce":77724832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6806","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6807","nonce":77724832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.13:6804","nonce":77724832},{"type":"v1","addr":"10.249.82.13:6805","nonce":77724832}]},"public_addr":"10.249.82.13:6801/77724832","cluster_addr":"10.249.82.13:6803/77724832","heartbeat_back_addr":"10.249.82.13:6807/77724832","heartbeat_front_addr":"10.249.82.13:6805/77724832","state":["exists","up"]},{"osd":1,"uuid":"d5614f90-f43f-4424-a246-23347f5c3bf5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6800","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6801","nonce":424166575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6802","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6803","nonce":424166575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6806","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6807","nonce":424166575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.15:6804","nonce":424166575},{"type":"v1","addr":"10.249.81.15:6805","nonce":424166575}]},"public_addr":"10.249.81.15:6801/424166575","cluster_addr":"10.249.81.15:6803/424166575","heartbeat_back_addr":"10.249.81.15:6807/424166575","heartbeat_front_addr":"10.249.81.15:6805/424166575","state":["exists","up"]},{"osd":2,"uuid":"1a7f1675-ec12-4c4f-af01-7e3a224c4265","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6800","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6801","nonce":550832532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6802","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6803","nonce":550832532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6806","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6807","nonce":550832532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.16:6804","nonce":550832532},{"type":"v1","addr":"10.249.81.16:6805","nonce":550832532}]},"public_addr":"10.249.81.16:6801/550832532","cluster_addr":"10.249.81.16:6803/550832532","heartbeat_back_addr":"10.249.81.16:6807/550832532","heartbeat_front_addr":"10.249.81.16:6805/550832532","state":["exists","up"]},{"osd":3,"uuid":"5a57fa32-a917-4784-a91e-2921e7218b40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6800","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6801","nonce":3508099279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6802","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6803","nonce":3508099279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6806","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6807","nonce":3508099279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.12:6804","nonce":3508099279},{"type":"v1","addr":"10.249.80.12:6805","nonce":3508099279}]},"public_addr":"10.249.80.12:6801/3508099279","cluster_addr":"10.249.80.12:6803/3508099279","heartbeat_back_addr":"10.249.80.12:6807/3508099279","heartbeat_front_addr":"10.249.80.12:6805/3508099279","state":["exists","up"]},{"osd":4,"uuid":"0a26bc99-8855-4473-be5a-78b35f554d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6800","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6801","nonce":3874051968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6802","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6803","nonce":3874051968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6806","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6807","nonce":3874051968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.13:6804","nonce":3874051968},{"type":"v1","addr":"10.249.80.13:6805","nonce":3874051968}]},"public_addr":"10.249.80.13:6801/3874051968","cluster_addr":"10.249.80.13:6803/3874051968","heartbeat_back_addr":"10.249.80.13:6807/3874051968","heartbeat_front_addr":"10.249.80.13:6805/3874051968","state":["exists","up"]},{"osd":5,"uuid":"1e82124e-368c-4fc5-9edc-bbeb2d81ff9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6800","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6801","nonce":2272372010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6802","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6803","nonce":2272372010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6806","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6807","nonce":2272372010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.17:6804","nonce":2272372010},{"type":"v1","addr":"10.249.81.17:6805","nonce":2272372010}]},"public_addr":"10.249.81.17:6801/2272372010","cluster_addr":"10.249.81.17:6803/2272372010","heartbeat_back_addr":"10.249.81.17:6807/2272372010","heartbeat_front_addr":"10.249.81.17:6805/2272372010","state":["exists","up"]},{"osd":6,"uuid":"525ee236-4220-4194-824c-777f2b3cc5a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6800","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6801","nonce":4147345364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6802","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6803","nonce":4147345364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6806","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6807","nonce":4147345364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.81.18:6804","nonce":4147345364},{"type":"v1","addr":"10.249.81.18:6805","nonce":4147345364}]},"public_addr":"10.249.81.18:6801/4147345364","cluster_addr":"10.249.81.18:6803/4147345364","heartbeat_back_addr":"10.249.81.18:6807/4147345364","heartbeat_front_addr":"10.249.81.18:6805/4147345364","state":["exists","up"]},{"osd":7,"uuid":"be39cf83-817f-4c1c-9c32-b1edad4a985c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6800","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6801","nonce":4147162602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6802","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6803","nonce":4147162602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6806","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6807","nonce":4147162602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.14:6804","nonce":4147162602},{"type":"v1","addr":"10.249.82.14:6805","nonce":4147162602}]},"public_addr":"10.249.82.14:6801/4147162602","cluster_addr":"10.249.82.14:6803/4147162602","heartbeat_back_addr":"10.249.82.14:6807/4147162602","heartbeat_front_addr":"10.249.82.14:6805/4147162602","state":["exists","up"]},{"osd":8,"uuid":"2b14b01b-a3e2-4e55-9532-768d8b53378d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6800","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6801","nonce":3981665281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6802","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6803","nonce":3981665281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6806","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6807","nonce":3981665281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.16:6804","nonce":3981665281},{"type":"v1","addr":"10.249.82.16:6805","nonce":3981665281}]},"public_addr":"10.249.82.16:6801/3981665281","cluster_addr":"10.249.82.16:6803/3981665281","heartbeat_back_addr":"10.249.82.16:6807/3981665281","heartbeat_front_addr":"10.249.82.16:6805/3981665281","state":["exists","up"]},{"osd":9,"uuid":"7494a9de-3a85-43c3-be44-e5ead46781ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6800","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6801","nonce":433732131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6802","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6803","nonce":433732131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6806","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6807","nonce":433732131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.82.15:6804","nonce":433732131},{"type":"v1","addr":"10.249.82.15:6805","nonce":433732131}]},"public_addr":"10.249.82.15:6801/433732131","cluster_addr":"10.249.82.15:6803/433732131","heartbeat_back_addr":"10.249.82.15:6807/433732131","heartbeat_front_addr":"10.249.82.15:6805/433732131","state":["exists","up"]},{"osd":10,"uuid":"e87e0787-5205-4211-b83a-263628693ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6800","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6801","nonce":3117418671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6802","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6803","nonce":3117418671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6806","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6807","nonce":3117418671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.15:6804","nonce":3117418671},{"type":"v1","addr":"10.249.80.15:6805","nonce":3117418671}]},"public_addr":"10.249.80.15:6801/3117418671","cluster_addr":"10.249.80.15:6803/3117418671","heartbeat_back_addr":"10.249.80.15:6807/3117418671","heartbeat_front_addr":"10.249.80.15:6805/3117418671","state":["exists","up"]},{"osd":11,"uuid":"79a411c2-f1c8-42c2-8484-607eedabfba7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6800","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6801","nonce":1997927166}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6802","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6803","nonce":1997927166}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6806","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6807","nonce":1997927166}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.80.16:6804","nonce":1997927166},{"type":"v1","addr":"10.249.80.16:6805","nonce":1997927166}]},"public_addr":"10.249.80.16:6801/1997927166","cluster_addr":"10.249.80.16:6803/1997927166","heartbeat_back_addr":"10.249.80.16:6807/1997927166","heartbeat_front_addr":"10.249.80.16:6805/1997927166","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:25.126118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:26.315593+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:30.036390+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:45.248011+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:43.033473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:40.469337+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:42.671520+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:48.203811+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:48.299728+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:49.486769+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:59.499523+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-21T14:56:59.399609+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.80.6:0/4223835587":"2022-03-22T14:55:13.010685+0000","10.249.80.6:6800/724391104":"2022-03-22T14:55:13.010685+0000","10.249.80.6:6801/724391104":"2022-03-22T14:55:13.010685+0000","10.249.80.6:0/1128621226":"2022-03-22T14:55:13.010685+0000","10.249.80.6:0/3353249299":"2022-03-22T14:55:13.010685+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-21T14:57:14.266 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.0 flush_pg_stats 2022-03-21T14:57:14.267 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.1 flush_pg_stats 2022-03-21T14:57:14.268 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.2 flush_pg_stats 2022-03-21T14:57:14.269 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.3 flush_pg_stats 2022-03-21T14:57:14.269 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.4 flush_pg_stats 2022-03-21T14:57:14.270 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.5 flush_pg_stats 2022-03-21T14:57:14.270 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.6 flush_pg_stats 2022-03-21T14:57:14.271 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.7 flush_pg_stats 2022-03-21T14:57:14.272 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.8 flush_pg_stats 2022-03-21T14:57:14.272 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.9 flush_pg_stats 2022-03-21T14:57:14.273 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.10 flush_pg_stats 2022-03-21T14:57:14.274 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph tell osd.11 flush_pg_stats 2022-03-21T14:57:14.837 INFO:teuthology.orchestra.run.smithi043.stdout:73014444043 2022-03-21T14:57:14.838 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.0 2022-03-21T14:57:14.999 INFO:teuthology.orchestra.run.smithi043.stdout:81604378637 2022-03-21T14:57:15.000 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.1 2022-03-21T14:57:15.268 INFO:teuthology.orchestra.run.smithi043.stdout:141733920772 2022-03-21T14:57:15.269 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.11 2022-03-21T14:57:15.310 INFO:teuthology.orchestra.run.smithi043.stdout:98784247816 2022-03-21T14:57:15.311 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.5 2022-03-21T14:57:15.314 INFO:teuthology.orchestra.run.smithi043.stdout:120259084295 2022-03-21T14:57:15.315 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.7 2022-03-21T14:57:15.319 INFO:teuthology.orchestra.run.smithi043.stdout:85899345930 2022-03-21T14:57:15.319 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.2 2022-03-21T14:57:15.330 INFO:teuthology.orchestra.run.smithi043.stdout:124554051591 2022-03-21T14:57:15.330 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.9 2022-03-21T14:57:15.332 INFO:teuthology.orchestra.run.smithi043.stdout:120259084295 2022-03-21T14:57:15.333 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.8 2022-03-21T14:57:15.334 INFO:teuthology.orchestra.run.smithi043.stdout:141733920773 2022-03-21T14:57:15.334 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.10 2022-03-21T14:57:15.337 INFO:teuthology.orchestra.run.smithi043.stdout:111669149703 2022-03-21T14:57:15.337 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.3 2022-03-21T14:57:15.375 INFO:teuthology.orchestra.run.smithi043.stdout:111669149706 2022-03-21T14:57:15.375 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.6 2022-03-21T14:57:15.451 INFO:teuthology.orchestra.run.smithi043.stdout:111669149706 2022-03-21T14:57:15.452 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.4 2022-03-21T14:57:15.852 INFO:teuthology.orchestra.run.smithi043.stdout:73014444042 2022-03-21T14:57:15.856 INFO:teuthology.orchestra.run.smithi043.stdout:81604378636 2022-03-21T14:57:16.027 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378637 got 81604378636 for osd.1 2022-03-21T14:57:16.034 INFO:teuthology.orchestra.run.smithi043.stdout:141733920771 2022-03-21T14:57:16.109 INFO:teuthology.orchestra.run.smithi043.stdout:85899345928 2022-03-21T14:57:16.120 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444043 got 73014444042 for osd.0 2022-03-21T14:57:16.124 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920772 got 141733920771 for osd.11 2022-03-21T14:57:16.156 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345930 got 85899345928 for osd.2 2022-03-21T14:57:16.173 INFO:teuthology.orchestra.run.smithi043.stdout:98784247815 2022-03-21T14:57:16.206 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247816 got 98784247815 for osd.5 2022-03-21T14:57:16.348 INFO:teuthology.orchestra.run.smithi043.stdout:111669149704 2022-03-21T14:57:16.369 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149706 got 111669149704 for osd.4 2022-03-21T14:57:16.393 INFO:teuthology.orchestra.run.smithi043.stdout:120259084294 2022-03-21T14:57:16.419 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084295 got 120259084294 for osd.7 2022-03-21T14:57:16.422 INFO:teuthology.orchestra.run.smithi043.stdout:124554051590 2022-03-21T14:57:16.432 INFO:teuthology.orchestra.run.smithi043.stdout:111669149705 2022-03-21T14:57:16.448 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051591 got 124554051590 for osd.9 2022-03-21T14:57:16.453 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149706 got 111669149705 for osd.6 2022-03-21T14:57:16.531 INFO:teuthology.orchestra.run.smithi043.stdout:111669149701 2022-03-21T14:57:16.551 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149703 got 111669149701 for osd.3 2022-03-21T14:57:16.565 INFO:teuthology.orchestra.run.smithi043.stdout:120259084294 2022-03-21T14:57:16.584 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084295 got 120259084294 for osd.8 2022-03-21T14:57:16.585 INFO:teuthology.orchestra.run.smithi043.stdout:141733920772 2022-03-21T14:57:16.633 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920773 got 141733920772 for osd.10 2022-03-21T14:57:17.028 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.1 2022-03-21T14:57:17.121 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.0 2022-03-21T14:57:17.125 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.11 2022-03-21T14:57:17.157 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.2 2022-03-21T14:57:17.207 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.5 2022-03-21T14:57:17.371 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.4 2022-03-21T14:57:17.420 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.7 2022-03-21T14:57:17.449 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.9 2022-03-21T14:57:17.454 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.6 2022-03-21T14:57:17.552 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.3 2022-03-21T14:57:17.586 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.8 2022-03-21T14:57:17.634 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph osd last-stat-seq osd.10 2022-03-21T14:57:17.697 INFO:teuthology.orchestra.run.smithi043.stdout:81604378637 2022-03-21T14:57:17.761 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378637 got 81604378637 for osd.1 2022-03-21T14:57:17.762 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:17.801 INFO:teuthology.orchestra.run.smithi043.stdout:73014444043 2022-03-21T14:57:17.847 INFO:teuthology.orchestra.run.smithi043.stdout:85899345930 2022-03-21T14:57:17.917 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444043 got 73014444043 for osd.0 2022-03-21T14:57:17.918 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.029 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345930 got 85899345930 for osd.2 2022-03-21T14:57:18.029 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.053 INFO:teuthology.orchestra.run.smithi043.stdout:141733920773 2022-03-21T14:57:18.106 INFO:teuthology.orchestra.run.smithi043.stdout:98784247817 2022-03-21T14:57:18.125 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920772 got 141733920773 for osd.11 2022-03-21T14:57:18.126 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.148 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247816 got 98784247817 for osd.5 2022-03-21T14:57:18.149 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.229 INFO:teuthology.orchestra.run.smithi043.stdout:111669149706 2022-03-21T14:57:18.294 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149706 got 111669149706 for osd.4 2022-03-21T14:57:18.294 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.299 INFO:teuthology.orchestra.run.smithi043.stdout:111669149706 2022-03-21T14:57:18.324 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149706 got 111669149706 for osd.6 2022-03-21T14:57:18.325 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.448 INFO:teuthology.orchestra.run.smithi043.stdout:120259084296 2022-03-21T14:57:18.449 INFO:teuthology.orchestra.run.smithi043.stdout:111669149703 2022-03-21T14:57:18.469 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149703 got 111669149703 for osd.3 2022-03-21T14:57:18.469 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.471 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084295 got 120259084296 for osd.7 2022-03-21T14:57:18.472 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.477 INFO:teuthology.orchestra.run.smithi043.stdout:124554051591 2022-03-21T14:57:18.495 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051591 got 124554051591 for osd.9 2022-03-21T14:57:18.495 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.537 INFO:teuthology.orchestra.run.smithi043.stdout:120259084296 2022-03-21T14:57:18.555 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084295 got 120259084296 for osd.8 2022-03-21T14:57:18.555 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.589 INFO:teuthology.orchestra.run.smithi043.stdout:141733920774 2022-03-21T14:57:18.607 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920773 got 141733920774 for osd.10 2022-03-21T14:57:18.607 DEBUG:teuthology.parallel:result is None 2022-03-21T14:57:18.608 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-21T14:57:18.608 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph pg dump --format=json 2022-03-21T14:57:19.193 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:19.196 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2022-03-21T14:57:19.213 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2022-03-21T14:57:17.723472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67524,"kb_used_data":5924,"kb_used_omap":0,"kb_used_meta":61248,"kb_avail":1124874300,"statfs":{"total":1151940427776,"available":1151871283200,"internally_reserved":0,"allocated":6066176,"data_stored":3559991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62717952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004249"},"pg_stats":[{"pgid":"1.0","version":"35'137","reported_seq":237,"reported_epoch":37,"state":"active+clean","last_fresh":"2022-03-21T14:57:05.247411+0000","last_change":"2022-03-21T14:57:05.247411+0000","last_active":"2022-03-21T14:57:05.247411+0000","last_peered":"2022-03-21T14:57:05.247411+0000","last_clean":"2022-03-21T14:57:05.247411+0000","last_became_active":"2022-03-21T14:57:05.246273+0000","last_became_peered":"2022-03-21T14:57:05.246273+0000","last_unstale":"2022-03-21T14:57:05.247411+0000","last_undegraded":"2022-03-21T14:57:05.247411+0000","last_fullsized":"2022-03-21T14:57:05.247411+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:56:40.411404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:56:40.411404+0000","last_clean_scrub_stamp":"2022-03-21T14:56:40.411404+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:15:15.086297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2068480,"data_stored":2054272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":11,"up_from":33,"seq":141733920773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5880,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739272,"statfs":{"total":95995035648,"available":95989014528,"internally_reserved":0,"allocated":901120,"data_stored":693020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.601}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7110000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3279999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.95}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0009999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]}]},{"osd":10,"up_from":33,"seq":141733920774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5236,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739916,"statfs":{"total":95995035648,"available":95989673984,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.351}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1020000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0470000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1630000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3420000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7210000000000001}]}]},{"osd":9,"up_from":29,"seq":124554051591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4119999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2669999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4809999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6879999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]}]},{"osd":8,"up_from":28,"seq":120259084296,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.776}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9279999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]}]},{"osd":7,"up_from":28,"seq":120259084296,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4980000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3540000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.722}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9870000000000001}]}]},{"osd":6,"up_from":26,"seq":111669149706,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7609999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.649}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1190000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7829999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1829999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5299999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9450000000000001}]}]},{"osd":1,"up_from":19,"seq":81604378638,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6652,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738500,"statfs":{"total":95995035648,"available":95988224000,"internally_reserved":0,"allocated":905216,"data_stored":693393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.206}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.694}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.597}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.452}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4289999999999998}]}]},{"osd":0,"up_from":17,"seq":73014444044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5364,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739788,"statfs":{"total":95995035648,"available":95989542912,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9710000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0709999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]}]},{"osd":2,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6652,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738500,"statfs":{"total":95995035648,"available":95988224000,"internally_reserved":0,"allocated":905216,"data_stored":693393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]}]},{"osd":5,"up_from":23,"seq":98784247817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5364,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739788,"statfs":{"total":95995035648,"available":95989542912,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0190000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9710000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.847}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2480000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.274}]}]},{"osd":3,"up_from":26,"seq":111669149703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8859999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4910000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.954}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]}]},{"osd":4,"up_from":26,"seq":111669149706,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5876,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93739276,"statfs":{"total":95995035648,"available":95989018624,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4390000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4279999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-21T14:57:19.215 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph pg dump --format=json 2022-03-21T14:57:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:19.762 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2022-03-21T14:57:19.781 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2022-03-21T14:57:19.724062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67524,"kb_used_data":5924,"kb_used_omap":0,"kb_used_meta":61248,"kb_avail":1124874300,"statfs":{"total":1151940427776,"available":1151871283200,"internally_reserved":0,"allocated":6066176,"data_stored":3559991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62717952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004283"},"pg_stats":[{"pgid":"1.0","version":"35'137","reported_seq":237,"reported_epoch":37,"state":"active+clean","last_fresh":"2022-03-21T14:57:05.247411+0000","last_change":"2022-03-21T14:57:05.247411+0000","last_active":"2022-03-21T14:57:05.247411+0000","last_peered":"2022-03-21T14:57:05.247411+0000","last_clean":"2022-03-21T14:57:05.247411+0000","last_became_active":"2022-03-21T14:57:05.246273+0000","last_became_peered":"2022-03-21T14:57:05.246273+0000","last_unstale":"2022-03-21T14:57:05.247411+0000","last_undegraded":"2022-03-21T14:57:05.247411+0000","last_fullsized":"2022-03-21T14:57:05.247411+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-21T14:56:40.411404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-21T14:56:40.411404+0000","last_clean_scrub_stamp":"2022-03-21T14:56:40.411404+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T21:15:15.086297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":467488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2068480,"data_stored":2054272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":11,"up_from":33,"seq":141733920773,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5880,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739272,"statfs":{"total":95995035648,"available":95989014528,"internally_reserved":0,"allocated":901120,"data_stored":693020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.601}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7110000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3279999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.95}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0009999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]}]},{"osd":10,"up_from":33,"seq":141733920774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5236,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739916,"statfs":{"total":95995035648,"available":95989673984,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.764}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.351}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1020000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0470000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1630000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3420000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7210000000000001}]}]},{"osd":9,"up_from":29,"seq":124554051592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.327}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0310000000000001}]}]},{"osd":8,"up_from":28,"seq":120259084296,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.776}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9279999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]}]},{"osd":7,"up_from":28,"seq":120259084296,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4980000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3540000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.722}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9870000000000001}]}]},{"osd":6,"up_from":26,"seq":111669149707,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6890000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9219999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.835}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9199999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.96}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9480000000000004}]}]},{"osd":1,"up_from":19,"seq":81604378638,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6652,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738500,"statfs":{"total":95995035648,"available":95988224000,"internally_reserved":0,"allocated":905216,"data_stored":693393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.206}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.694}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.597}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.452}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4289999999999998}]}]},{"osd":0,"up_from":17,"seq":73014444044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5364,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739788,"statfs":{"total":95995035648,"available":95989542912,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9710000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0709999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]}]},{"osd":2,"up_from":20,"seq":85899345931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6652,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738500,"statfs":{"total":95995035648,"available":95988224000,"internally_reserved":0,"allocated":905216,"data_stored":693393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1680000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0129999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2440000000000002}]}]},{"osd":5,"up_from":23,"seq":98784247817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5364,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739788,"statfs":{"total":95995035648,"available":95989542912,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0190000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9710000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.847}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2480000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.274}]}]},{"osd":3,"up_from":26,"seq":111669149704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5300,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739852,"statfs":{"total":95995035648,"available":95989608448,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5980000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]}]},{"osd":4,"up_from":26,"seq":111669149707,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5876,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93739276,"statfs":{"total":95995035648,"available":95989018624,"internally_reserved":0,"allocated":372736,"data_stored":164465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5389999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7829999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0019999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-21T14:57:19.783 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-21T14:57:19.783 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-21T14:57:19.784 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-21T14:57:19.784 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph health --format=json 2022-03-21T14:57:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:20.476 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-21T14:57:20.478 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-21T14:57:20.478 INFO:tasks.rook:Rook complete, yielding 2022-03-21T14:57:20.478 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-21T14:57:20.489 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph -s 2022-03-21T14:57:21.091 INFO:teuthology.orchestra.run.smithi043.stdout: cluster: 2022-03-21T14:57:21.091 INFO:teuthology.orchestra.run.smithi043.stdout: id: 39ceab48-33b1-458f-a341-29eb1474d30e 2022-03-21T14:57:21.092 INFO:teuthology.orchestra.run.smithi043.stdout: health: HEALTH_OK 2022-03-21T14:57:21.092 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:21.092 INFO:teuthology.orchestra.run.smithi043.stdout: services: 2022-03-21T14:57:21.092 INFO:teuthology.orchestra.run.smithi043.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-21T14:57:21.093 INFO:teuthology.orchestra.run.smithi043.stdout: mgr: a(active, since 2m) 2022-03-21T14:57:21.093 INFO:teuthology.orchestra.run.smithi043.stdout: osd: 12 osds: 12 up (since 20s), 12 in (since 63s) 2022-03-21T14:57:21.093 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:21.093 INFO:teuthology.orchestra.run.smithi043.stdout: data: 2022-03-21T14:57:21.094 INFO:teuthology.orchestra.run.smithi043.stdout: pools: 1 pools, 1 pgs 2022-03-21T14:57:21.094 INFO:teuthology.orchestra.run.smithi043.stdout: objects: 2 objects, 577 KiB 2022-03-21T14:57:21.094 INFO:teuthology.orchestra.run.smithi043.stdout: usage: 66 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-21T14:57:21.095 INFO:teuthology.orchestra.run.smithi043.stdout: pgs: 1 active+clean 2022-03-21T14:57:21.095 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:57:21.111 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status 2022-03-21T14:57:21.800 INFO:teuthology.orchestra.run.smithi043.stdout:Backend: rook 2022-03-21T14:57:21.801 INFO:teuthology.orchestra.run.smithi043.stdout:Available: Yes 2022-03-21T14:57:21.826 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ps 2022-03-21T14:57:22.401 INFO:teuthology.orchestra.run.smithi043.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-21T14:57:22.402 INFO:teuthology.orchestra.run.smithi043.stdout:crashcollector.crash smithi043 running (74s) 0s ago 74s - - 7c0c85ada2e4 2022-03-21T14:57:22.402 INFO:teuthology.orchestra.run.smithi043.stdout:crashcollector.crash smithi065 running (2m) 0s ago 2m - - 7c0c85ada2e4 2022-03-21T14:57:22.402 INFO:teuthology.orchestra.run.smithi043.stdout:crashcollector.crash smithi154 running (2m) 0s ago 2m - - 7c0c85ada2e4 2022-03-21T14:57:22.403 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.a smithi043 running (2m) 0s ago 2m - - 7c0c85ada2e4 2022-03-21T14:57:22.403 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a smithi154 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-21T14:57:22.403 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b smithi065 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-21T14:57:22.403 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c smithi043 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:57:22.404 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 smithi154 running (80s) 0s ago 81s - - 7c0c85ada2e4 2022-03-21T14:57:22.404 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 smithi065 running (78s) 0s ago 78s - - 7c0c85ada2e4 2022-03-21T14:57:22.404 INFO:teuthology.orchestra.run.smithi043.stdout:osd.10 smithi043 running (56s) 0s ago 57s - - 7c0c85ada2e4 2022-03-21T14:57:22.404 INFO:teuthology.orchestra.run.smithi043.stdout:osd.11 smithi043 running (53s) 0s ago 54s - - 7c0c85ada2e4 2022-03-21T14:57:22.404 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 smithi065 running (76s) 0s ago 76s - - 7c0c85ada2e4 2022-03-21T14:57:22.405 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 smithi043 running (74s) 0s ago 75s - - 7c0c85ada2e4 2022-03-21T14:57:22.405 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 smithi043 running (74s) 0s ago 74s - - 7c0c85ada2e4 2022-03-21T14:57:22.405 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 smithi065 running (65s) 0s ago 65s - - 7c0c85ada2e4 2022-03-21T14:57:22.405 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 smithi065 running (61s) 0s ago 61s - - 7c0c85ada2e4 2022-03-21T14:57:22.406 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 smithi154 running (60s) 0s ago 60s - - 7c0c85ada2e4 2022-03-21T14:57:22.406 INFO:teuthology.orchestra.run.smithi043.stdout:osd.8 smithi154 running (57s) 0s ago 57s - - 7c0c85ada2e4 2022-03-21T14:57:22.406 INFO:teuthology.orchestra.run.smithi043.stdout:osd.9 smithi154 running (59s) 0s ago 59s - - 7c0c85ada2e4 2022-03-21T14:57:22.427 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls 2022-03-21T14:57:23.145 INFO:teuthology.orchestra.run.smithi043.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-21T14:57:23.145 INFO:teuthology.orchestra.run.smithi043.stdout:crash 3/3 0s ago 2m * 2022-03-21T14:57:23.146 INFO:teuthology.orchestra.run.smithi043.stdout:mgr 1/1 0s ago 2m count:1 2022-03-21T14:57:23.146 INFO:teuthology.orchestra.run.smithi043.stdout:mon 3/3 0s ago 5m count:3 2022-03-21T14:57:23.146 INFO:teuthology.orchestra.run.smithi043.stdout:osd 24 0s ago 82s 2022-03-21T14:57:23.146 INFO:teuthology.orchestra.run.smithi043.stdout:osd.all-available-devices 0 0s ago - * 2022-03-21T14:57:23.171 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch device ls 2022-03-21T14:57:23.799 INFO:teuthology.orchestra.run.smithi043.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-21T14:57:23.799 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-21T14:57:23.800 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-21T14:57:23.800 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-21T14:57:23.800 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-21T14:57:23.800 INFO:teuthology.orchestra.run.smithi043.stdout:smithi065 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-21T14:57:23.800 INFO:teuthology.orchestra.run.smithi043.stdout:smithi065 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-21T14:57:23.801 INFO:teuthology.orchestra.run.smithi043.stdout:smithi065 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-21T14:57:23.801 INFO:teuthology.orchestra.run.smithi043.stdout:smithi065 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-21T14:57:23.801 INFO:teuthology.orchestra.run.smithi043.stdout:smithi154 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-21T14:57:23.801 INFO:teuthology.orchestra.run.smithi043.stdout:smithi154 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-21T14:57:23.801 INFO:teuthology.orchestra.run.smithi043.stdout:smithi154 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-21T14:57:23.802 INFO:teuthology.orchestra.run.smithi043.stdout:smithi154 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-21T14:57:23.822 INFO:teuthology.run_tasks:Running task exec... 2022-03-21T14:57:23.832 INFO:teuthology.task.exec:Executing custom commands... 2022-03-21T14:57:23.833 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi043.front.sepia.ceph.com 2022-03-21T14:57:23.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-21T14:57:23.834 DEBUG:teuthology.orchestra.run.smithi043:> toolbox() { 2022-03-21T14:57:23.834 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-21T14:57:23.834 DEBUG:teuthology.orchestra.run.smithi043:> } 2022-03-21T14:57:23.834 DEBUG:teuthology.orchestra.run.smithi043:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-21T14:57:23.835 DEBUG:teuthology.orchestra.run.smithi043:> toolbox ceph orch osd rm 0 --force 2022-03-21T14:57:23.835 DEBUG:teuthology.orchestra.run.smithi043:> removed_pv="" 2022-03-21T14:57:23.835 DEBUG:teuthology.orchestra.run.smithi043:> while [ "$removed_pv" = "" ] 2022-03-21T14:57:23.836 DEBUG:teuthology.orchestra.run.smithi043:> do 2022-03-21T14:57:23.836 DEBUG:teuthology.orchestra.run.smithi043:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-21T14:57:23.836 DEBUG:teuthology.orchestra.run.smithi043:> sleep 3s 2022-03-21T14:57:23.836 DEBUG:teuthology.orchestra.run.smithi043:> done 2022-03-21T14:57:23.837 DEBUG:teuthology.orchestra.run.smithi043:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-21T14:57:23.837 DEBUG:teuthology.orchestra.run.smithi043:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-21T14:57:23.837 DEBUG:teuthology.orchestra.run.smithi043:> toolbox ceph orch device zap $host $target_path --force 2022-03-21T14:57:23.837 DEBUG:teuthology.orchestra.run.smithi043:> zap_completion="0" 2022-03-21T14:57:23.838 DEBUG:teuthology.orchestra.run.smithi043:> while [ "$zap_completion" = "0" ] 2022-03-21T14:57:23.838 DEBUG:teuthology.orchestra.run.smithi043:> do 2022-03-21T14:57:23.838 DEBUG:teuthology.orchestra.run.smithi043:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-21T14:57:23.839 DEBUG:teuthology.orchestra.run.smithi043:> sleep 3s 2022-03-21T14:57:23.839 DEBUG:teuthology.orchestra.run.smithi043:> done 2022-03-21T14:57:23.839 DEBUG:teuthology.orchestra.run.smithi043:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-21T14:57:23.840 DEBUG:teuthology.orchestra.run.smithi043:> toolbox ceph orch apply osd --all-available-devices 2022-03-21T14:57:23.840 DEBUG:teuthology.orchestra.run.smithi043:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-21T14:57:23.840 DEBUG:teuthology.orchestra.run.smithi043:> num_osd="0" 2022-03-21T14:57:23.841 DEBUG:teuthology.orchestra.run.smithi043:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-21T14:57:23.841 DEBUG:teuthology.orchestra.run.smithi043:> do 2022-03-21T14:57:23.841 DEBUG:teuthology.orchestra.run.smithi043:> echo "waiting for osd to come back up" 2022-03-21T14:57:23.842 DEBUG:teuthology.orchestra.run.smithi043:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-21T14:57:23.842 DEBUG:teuthology.orchestra.run.smithi043:> sleep 30s 2022-03-21T14:57:23.842 DEBUG:teuthology.orchestra.run.smithi043:> done 2022-03-21T14:57:23.842 DEBUG:teuthology.orchestra.run.smithi043:> ' 2022-03-21T14:57:23.854 INFO:teuthology.orchestra.run.smithi043.stderr:++ toolbox ceph osd stat 2022-03-21T14:57:23.854 INFO:teuthology.orchestra.run.smithi043.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-21T14:57:23.855 INFO:teuthology.orchestra.run.smithi043.stderr:++ cut -f3 -d ' ' 2022-03-21T14:57:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-21T14:57:27.270 INFO:teuthology.orchestra.run.smithi043.stderr:+ orig_num_osd=12 2022-03-21T14:57:27.270 INFO:teuthology.orchestra.run.smithi043.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-21T14:57:27.271 INFO:teuthology.orchestra.run.smithi043.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-21T14:57:27.341 INFO:teuthology.orchestra.run.smithi043.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-21T14:57:27.995 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:27.993654 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-21T14:57:27.996 INFO:tasks.rook.operator.smithi043.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-21T14:57:27.996 INFO:tasks.rook.operator.smithi043.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-21T14:57:27.996 INFO:tasks.rook.operator.smithi043.stdout:   ... // 3 identical fields 2022-03-21T14:57:27.997 INFO:tasks.rook.operator.smithi043.stdout:   Config: nil, 2022-03-21T14:57:27.997 INFO:tasks.rook.operator.smithi043.stdout:   Selection: {}, 2022-03-21T14:57:27.997 INFO:tasks.rook.operator.smithi043.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-21T14:57:27.997 INFO:tasks.rook.operator.smithi043.stdout:   ... // 6 identical elements 2022-03-21T14:57:27.997 INFO:tasks.rook.operator.smithi043.stdout:   {Name: "smithi065-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi065-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-21T14:57:27.998 INFO:tasks.rook.operator.smithi043.stdout:   {Name: "smithi065-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi065-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-21T14:57:27.998 INFO:tasks.rook.operator.smithi043.stdout:-  { 2022-03-21T14:57:27.998 INFO:tasks.rook.operator.smithi043.stdout:-  Name: "smithi154-lv-1", 2022-03-21T14:57:27.998 INFO:tasks.rook.operator.smithi043.stdout:-  Count: 1, 2022-03-21T14:57:27.999 INFO:tasks.rook.operator.smithi043.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-21T14:57:27.999 INFO:tasks.rook.operator.smithi043.stdout:-  }, 2022-03-21T14:57:27.999 INFO:tasks.rook.operator.smithi043.stdout:   {Name: "smithi154-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi154-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-21T14:57:27.999 INFO:tasks.rook.operator.smithi043.stdout:   {Name: "smithi154-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi154-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-21T14:57:27.999 INFO:tasks.rook.operator.smithi043.stdout:   {Name: "smithi154-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi154-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-21T14:57:28.000 INFO:tasks.rook.operator.smithi043.stdout:   }, 2022-03-21T14:57:28.000 INFO:tasks.rook.operator.smithi043.stdout:   }, 2022-03-21T14:57:28.000 INFO:tasks.rook.operator.smithi043.stdout:   Annotations: nil, 2022-03-21T14:57:28.000 INFO:tasks.rook.operator.smithi043.stdout:   Labels: nil, 2022-03-21T14:57:28.001 INFO:tasks.rook.operator.smithi043.stdout:   ... // 20 identical fields 2022-03-21T14:57:28.001 INFO:tasks.rook.operator.smithi043.stdout:  } 2022-03-21T14:57:28.001 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:27.993739 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-21T14:57:28.001 INFO:tasks.rook.operator.smithi043.stdout:I0321 14:57:27.993994 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-21T14:57:28.001 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:27.994040 I | operator: successfully started the controller-runtime manager 2022-03-21T14:57:28.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.023149 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-21T14:57:28.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.023176 I | operator: watching all namespaces for Ceph CRs 2022-03-21T14:57:28.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.023215 I | operator: setting up schemes 2022-03-21T14:57:28.029 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.030250 I | operator: setting up the controller-runtime manager 2022-03-21T14:57:28.233 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.234185 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-21T14:57:28.286 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286116 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-21T14:57:28.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286154 I | ceph-cluster-controller: successfully started 2022-03-21T14:57:28.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286209 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-21T14:57:28.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286229 I | ceph-crashcollector-controller: successfully started 2022-03-21T14:57:28.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286250 I | ceph-block-pool-controller: successfully started 2022-03-21T14:57:28.288 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286272 I | ceph-object-store-user-controller: successfully started 2022-03-21T14:57:28.288 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286292 I | ceph-object-realm-controller: successfully started 2022-03-21T14:57:28.288 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286306 I | ceph-object-zonegroup-controller: successfully started 2022-03-21T14:57:28.288 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286318 I | ceph-object-zone-controller: successfully started 2022-03-21T14:57:28.288 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286531 I | ceph-object-controller: successfully started 2022-03-21T14:57:28.289 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286599 I | ceph-file-controller: successfully started 2022-03-21T14:57:28.289 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286668 I | ceph-nfs-controller: successfully started 2022-03-21T14:57:28.289 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286725 I | ceph-rbd-mirror-controller: successfully started 2022-03-21T14:57:28.289 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286779 I | ceph-client-controller: successfully started 2022-03-21T14:57:28.290 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286820 I | ceph-filesystem-mirror-controller: successfully started 2022-03-21T14:57:28.290 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286867 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-21T14:57:28.290 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286906 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-21T14:57:28.290 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286946 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-21T14:57:28.290 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.286985 I | ceph-bucket-topic: successfully started 2022-03-21T14:57:28.291 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.287017 I | ceph-bucket-notification: successfully started 2022-03-21T14:57:28.291 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.287042 I | ceph-bucket-notification: successfully started 2022-03-21T14:57:28.291 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.287070 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-21T14:57:28.296 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.288658 I | operator: starting the controller-runtime manager 2022-03-21T14:57:28.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.394831 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-21T14:57:28.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.394875 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-21T14:57:28.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.394886 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-21T14:57:28.395 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.394896 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:57:28.480 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.481662 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-21T14:57:28.504 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.505305 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:57:28.504 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.505379 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-21T14:57:28.505 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.505391 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-21T14:57:28.505 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.505658 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:57:28.509 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.506997 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-21T14:57:28.509 INFO:tasks.rook.operator.smithi043.stdout:I0321 14:57:28.507081 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-21T14:57:28.544 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.545328 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-21T14:57:28.554 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.555612 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-21T14:57:28.619 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620274 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-21T14:57:28.620 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620305 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-21T14:57:28.620 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620320 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-21T14:57:28.620 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620332 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-21T14:57:28.620 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620344 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-21T14:57:28.621 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620358 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-21T14:57:28.622 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620367 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-21T14:57:28.622 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620376 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-21T14:57:28.622 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620385 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-21T14:57:28.622 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620394 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-21T14:57:28.623 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620404 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-21T14:57:28.623 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620428 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-21T14:57:28.623 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620439 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-21T14:57:28.623 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620451 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-21T14:57:28.624 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620464 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-21T14:57:28.624 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620478 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-21T14:57:28.624 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620558 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:57:28.624 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:28.620583 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:57:32.412 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:32.412689 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-21T14:57:32.549 INFO:teuthology.orchestra.run.smithi043.stdout:Success 2022-03-21T14:57:32.576 INFO:teuthology.orchestra.run.smithi043.stderr:+ removed_pv= 2022-03-21T14:57:32.576 INFO:teuthology.orchestra.run.smithi043.stderr:+ '[' '' = '' ']' 2022-03-21T14:57:32.577 INFO:teuthology.orchestra.run.smithi043.stderr:++ kubectl get pv 2022-03-21T14:57:32.577 INFO:teuthology.orchestra.run.smithi043.stderr:++ grep Released 2022-03-21T14:57:32.577 INFO:teuthology.orchestra.run.smithi043.stderr:++ cut -f1 -d ' ' 2022-03-21T14:57:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:+ removed_pv=smithi154-lv-1 2022-03-21T14:57:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:+ sleep 3s 2022-03-21T14:57:34.398 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398697 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-21T14:57:34.399 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398736 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-21T14:57:34.399 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398748 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-21T14:57:34.399 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398756 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-21T14:57:34.399 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398764 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-21T14:57:34.400 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398774 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-21T14:57:34.400 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398781 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-21T14:57:34.400 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398799 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-21T14:57:34.401 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398802 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-21T14:57:34.401 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398804 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-21T14:57:34.401 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398807 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-21T14:57:34.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398810 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-21T14:57:34.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398812 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-21T14:57:34.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398815 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-21T14:57:34.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398818 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-21T14:57:34.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398821 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-21T14:57:34.403 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398824 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-21T14:57:34.403 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398826 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-21T14:57:34.403 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398829 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-21T14:57:34.403 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398832 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-21T14:57:34.403 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398835 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-21T14:57:34.404 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398838 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-21T14:57:34.404 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398841 I | ceph-csi: Kubernetes version is 1.23 2022-03-21T14:57:34.404 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398844 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-21T14:57:34.404 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.398847 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-21T14:57:34.453 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.454791 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-21T14:57:34.465 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.463848 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:57:34.466 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.463889 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:57:34.466 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.463906 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:57:34.466 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.463914 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:57:34.466 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.463920 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:57:34.467 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.463926 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:57:34.467 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.463938 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-21T14:57:34.644 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.645399 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:57:34.645 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.645430 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:57:34.645 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.645439 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-21T14:57:34.732 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.732842 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-21T14:57:34.921 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.921426 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:57:34.921 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.921458 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:57:34.921 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:34.921465 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-21T14:57:35.005 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:35.005605 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:57:35.005 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:35.005622 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:57:35.005 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:35.005627 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-21T14:57:35.043 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:35.043832 I | ceph-csi: successfully started CSI CephFS driver 2022-03-21T14:57:35.221 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:35.222551 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-21T14:57:35.226 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:35.227016 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-21T14:57:35.226 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:35.227033 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-21T14:57:35.230 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:35.231415 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-21T14:57:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:+ '[' smithi154-lv-1 = '' ']' 2022-03-21T14:57:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:++ kubectl get pv smithi154-lv-1 -o 'jsonpath={.spec.local.path}' 2022-03-21T14:57:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-03-21T14:57:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:++ echo smithi154-lv-1 2022-03-21T14:57:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:++ cut -f1 -d - 2022-03-21T14:57:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:+ host=smithi154 2022-03-21T14:57:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:+ toolbox ceph orch device zap smithi154 /dev/vg_nvme/lv_1 --force 2022-03-21T14:57:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi154 /dev/vg_nvme/lv_1 --force 2022-03-21T14:57:35.977 INFO:teuthology.orchestra.run.smithi043.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-21T14:57:36.761 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/vg_nvme/lv_1 on smithi154 zapped 2022-03-21T14:57:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:+ zap_completion=0 2022-03-21T14:57:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:+ '[' 0 = 0 ']' 2022-03-21T14:57:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-21T14:57:36.920 INFO:teuthology.orchestra.run.smithi043.stderr:+ zap_completion= 2022-03-21T14:57:36.921 INFO:teuthology.orchestra.run.smithi043.stderr:+ sleep 3s 2022-03-21T14:57:37.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:37.575746 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-21T14:57:37.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:37.575776 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-21T14:57:37.580 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:37.581293 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:57:37.962 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:37.962821 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:57:37.962 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:37.963027 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:57:38.377 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:38.377668 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-21T14:57:38.555 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:38.555925 I | op-mon: start running mons 2022-03-21T14:57:38.759 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:38.760177 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:57:39.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:39.210020 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.111.185.43:6789","10.97.100.133:6789","10.96.127.40:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:57:39.288 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:39.288281 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:57:39.292 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:39.288582 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:57:39.922 INFO:teuthology.orchestra.run.smithi043.stderr:+ '[' '' = 0 ']' 2022-03-21T14:57:39.922 INFO:teuthology.orchestra.run.smithi043.stderr:+ kubectl patch pv smithi154-lv-1 -p '{"spec":{"claimRef": null}}' 2022-03-21T14:57:39.996 INFO:teuthology.orchestra.run.smithi043.stdout:persistentvolume/smithi154-lv-1 patched 2022-03-21T14:57:39.999 INFO:teuthology.orchestra.run.smithi043.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-21T14:57:39.999 INFO:teuthology.orchestra.run.smithi043.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-21T14:57:40.067 INFO:teuthology.orchestra.run.smithi043.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-21T14:57:40.208 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:40.208930 I | op-mon: targeting the mon count 3 2022-03-21T14:57:40.216 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:40.217072 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:57:40.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:40.559200 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:57:40.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:40.559223 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:57:41.215 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.215022 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:57:41.215 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.215056 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:57:41.333 INFO:teuthology.orchestra.run.smithi043.stdout:['Success'] 2022-03-21T14:57:41.341 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.341984 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-21T14:57:41.342 INFO:tasks.rook.operator.smithi043.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-21T14:57:41.342 INFO:tasks.rook.operator.smithi043.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-21T14:57:41.342 INFO:tasks.rook.operator.smithi043.stdout:   ... // 3 identical fields 2022-03-21T14:57:41.342 INFO:tasks.rook.operator.smithi043.stdout:   Config: nil, 2022-03-21T14:57:41.343 INFO:tasks.rook.operator.smithi043.stdout:   Selection: {}, 2022-03-21T14:57:41.344 INFO:tasks.rook.operator.smithi043.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-21T14:57:41.345 INFO:tasks.rook.operator.smithi043.stdout:   ... // 9 identical elements 2022-03-21T14:57:41.345 INFO:tasks.rook.operator.smithi043.stdout:   {Name: "smithi154-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi154-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-21T14:57:41.345 INFO:tasks.rook.operator.smithi043.stdout:   {Name: "smithi154-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi154-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-21T14:57:41.345 INFO:tasks.rook.operator.smithi043.stdout:+  { 2022-03-21T14:57:41.346 INFO:tasks.rook.operator.smithi043.stdout:+  Name: "smithi154-lv-1", 2022-03-21T14:57:41.346 INFO:tasks.rook.operator.smithi043.stdout:+  Count: 1, 2022-03-21T14:57:41.346 INFO:tasks.rook.operator.smithi043.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-21T14:57:41.346 INFO:tasks.rook.operator.smithi043.stdout:+  }, 2022-03-21T14:57:41.346 INFO:tasks.rook.operator.smithi043.stdout:   }, 2022-03-21T14:57:41.347 INFO:tasks.rook.operator.smithi043.stdout:   }, 2022-03-21T14:57:41.347 INFO:tasks.rook.operator.smithi043.stdout:   Annotations: nil, 2022-03-21T14:57:41.347 INFO:tasks.rook.operator.smithi043.stdout:   Labels: nil, 2022-03-21T14:57:41.347 INFO:tasks.rook.operator.smithi043.stdout:   ... // 20 identical fields 2022-03-21T14:57:41.348 INFO:tasks.rook.operator.smithi043.stdout:  } 2022-03-21T14:57:41.348 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.342106 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-21T14:57:41.348 INFO:tasks.rook.operator.smithi043.stdout:I0321 14:57:41.342328 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-21T14:57:41.358 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.358930 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-21T14:57:41.358 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.358965 I | operator: watching all namespaces for Ceph CRs 2022-03-21T14:57:41.359 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.359010 I | operator: setting up schemes 2022-03-21T14:57:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-21T14:57:41.362 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.362579 I | operator: setting up the controller-runtime manager 2022-03-21T14:57:41.436 INFO:teuthology.orchestra.run.smithi043.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-21T14:57:41.476 INFO:teuthology.orchestra.run.smithi043.stderr:+ num_osd=0 2022-03-21T14:57:41.476 INFO:teuthology.orchestra.run.smithi043.stderr:+ '[' 0 '!=' 12 ']' 2022-03-21T14:57:41.477 INFO:teuthology.orchestra.run.smithi043.stdout:waiting for osd to come back up 2022-03-21T14:57:41.478 INFO:teuthology.orchestra.run.smithi043.stderr:+ echo 'waiting for osd to come back up' 2022-03-21T14:57:41.478 INFO:teuthology.orchestra.run.smithi043.stderr:++ toolbox ceph osd stat 2022-03-21T14:57:41.478 INFO:teuthology.orchestra.run.smithi043.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-21T14:57:41.479 INFO:teuthology.orchestra.run.smithi043.stderr:++ cut -f3 -d ' ' 2022-03-21T14:57:41.566 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.566536 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-21T14:57:41.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.567080 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:57:41.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.567107 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:57:41.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.567206 W | op-mon: failed to set Rook and/or user-defined Ceph config options before starting mons; will retry after starting mons. failed to apply logging configuration: failed to set one or more Ceph configs: failed to set ceph config in the centralized mon configuration database; you may need to use the rook-config-override ConfigMap. output: : context canceled 2022-03-21T14:57:41.571 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.567224 I | op-mon: checking for basic quorum with existing mons 2022-03-21T14:57:41.571 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569560 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-21T14:57:41.571 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569580 I | ceph-cluster-controller: successfully started 2022-03-21T14:57:41.572 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569627 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-21T14:57:41.572 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569647 I | ceph-crashcollector-controller: successfully started 2022-03-21T14:57:41.572 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569669 I | ceph-block-pool-controller: successfully started 2022-03-21T14:57:41.572 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569688 I | ceph-object-store-user-controller: successfully started 2022-03-21T14:57:41.572 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569702 I | ceph-object-realm-controller: successfully started 2022-03-21T14:57:41.573 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569710 I | ceph-object-zonegroup-controller: successfully started 2022-03-21T14:57:41.573 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569719 I | ceph-object-zone-controller: successfully started 2022-03-21T14:57:41.573 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569804 I | ceph-object-controller: successfully started 2022-03-21T14:57:41.573 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569830 I | ceph-file-controller: successfully started 2022-03-21T14:57:41.574 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569849 I | ceph-nfs-controller: successfully started 2022-03-21T14:57:41.574 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569865 I | ceph-rbd-mirror-controller: successfully started 2022-03-21T14:57:41.574 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569884 I | ceph-client-controller: successfully started 2022-03-21T14:57:41.574 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569895 I | ceph-filesystem-mirror-controller: successfully started 2022-03-21T14:57:41.574 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569921 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-21T14:57:41.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569939 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-21T14:57:41.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569956 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-21T14:57:41.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569968 I | ceph-bucket-topic: successfully started 2022-03-21T14:57:41.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569980 I | ceph-bucket-notification: successfully started 2022-03-21T14:57:41.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569988 I | ceph-bucket-notification: successfully started 2022-03-21T14:57:41.576 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.569994 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-21T14:57:41.578 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.579016 I | operator: starting the controller-runtime manager 2022-03-21T14:57:41.579 INFO:teuthology.orchestra.run.smithi043.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-21T14:57:41.591 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.587829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-03-21T14:57:41.591 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.588535 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:57:41.613 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.613709 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:57:41.628 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.628849 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-03-21 14:57:41.620429934 +0000 UTC m=+416.488085770 LastTransitionTime:2022-03-21 14:57:41.620429731 +0000 UTC m=+416.488085594}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T14:57:41.628 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.628868 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-21T14:57:41.687 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.685305 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:57:41.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.687517 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-21T14:57:41.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.687586 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-21T14:57:41.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.687627 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-21T14:57:41.713 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:41.713353 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-21T14:57:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:+ num_osd=11 2022-03-21T14:57:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:+ sleep 30s 2022-03-21T14:57:42.206 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:42.206257 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:57:42.206 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:42.206306 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-21T14:57:42.206 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:42.206314 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-21T14:57:42.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:42.207467 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-21T14:57:42.209 INFO:tasks.rook.operator.smithi043.stdout:I0321 14:57:42.207544 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-21T14:57:42.406 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:42.406686 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:57:42.430 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:42.430507 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-21T14:57:42.714 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:42.714815 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-21T14:57:42.809 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:42.809388 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-21T14:57:43.074 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.074441 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-21T14:57:43.083 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082235 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-21T14:57:43.084 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082262 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-21T14:57:43.084 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082271 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-21T14:57:43.085 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082277 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-21T14:57:43.085 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082285 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-21T14:57:43.085 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082296 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-21T14:57:43.086 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082309 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-21T14:57:43.086 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082323 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-21T14:57:43.087 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082332 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-21T14:57:43.087 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082340 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-21T14:57:43.088 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082347 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-21T14:57:43.088 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082356 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-21T14:57:43.089 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082366 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-21T14:57:43.089 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082373 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-21T14:57:43.089 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082382 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-21T14:57:43.090 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082401 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-21T14:57:43.090 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082471 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:57:43.091 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:43.082497 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:57:46.489 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:46.489480 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-21T14:57:49.037 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035878 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-21T14:57:49.038 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035905 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-21T14:57:49.038 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035911 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-21T14:57:49.038 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035914 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-21T14:57:49.038 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035917 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-21T14:57:49.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035919 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-21T14:57:49.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035922 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-21T14:57:49.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035924 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-21T14:57:49.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035927 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-21T14:57:49.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035929 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-21T14:57:49.040 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035931 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-21T14:57:49.040 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035934 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-21T14:57:49.040 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035938 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-21T14:57:49.040 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035944 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-21T14:57:49.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035949 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-21T14:57:49.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035952 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-21T14:57:49.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035955 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-21T14:57:49.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035957 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-21T14:57:49.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035960 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-21T14:57:49.042 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035963 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-21T14:57:49.043 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035969 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-21T14:57:49.043 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035974 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-21T14:57:49.043 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035978 I | ceph-csi: Kubernetes version is 1.23 2022-03-21T14:57:49.043 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.035983 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-21T14:57:49.044 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.036002 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-21T14:57:49.044 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.040174 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-21T14:57:49.047 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.047206 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:57:49.047 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.047232 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:57:49.048 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.047238 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:57:49.048 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.047242 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:57:49.048 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.047247 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:57:49.048 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.047251 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:57:49.048 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.047255 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-21T14:57:49.171 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.171181 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:57:49.171 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.171212 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:57:49.171 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.171220 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-21T14:57:49.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.231856 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-21T14:57:49.476 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.476532 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-21T14:57:49.477 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.476559 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-21T14:57:49.477 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.476566 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-21T14:57:49.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.555847 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-21T14:57:49.561 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.556192 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-21T14:57:49.561 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.556221 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-21T14:57:49.561 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.556231 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-21T14:57:49.647 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.647350 I | ceph-csi: successfully started CSI CephFS driver 2022-03-21T14:57:49.871 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.870709 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-21T14:57:49.882 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.882646 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-21T14:57:49.883 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.882678 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-21T14:57:49.893 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:49.893660 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-21T14:57:52.563 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:57:52.563189 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-21T14:58:00.051 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.050503 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-21T14:58:00.051 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.050535 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-21T14:58:00.214 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.213427 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-21T14:58:00.214 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.213454 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-21T14:58:00.219 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.215289 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:58:00.725 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.725045 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:58:00.731 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.725501 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:58:00.731 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.725876 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:58:00.731 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.728202 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:58:00.732 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:00.728442 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:58:01.185 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:01.184756 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-21T14:58:01.223 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:01.223072 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-21T14:58:01.334 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:01.334368 I | op-mon: start running mons 2022-03-21T14:58:01.365 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:01.365073 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:58:01.369 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:01.365919 E | ceph-cluster-controller: failed to update cluster "rook-ceph" version. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T14:58:01.875 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:01.874989 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789","10.111.185.43:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:58:02.361 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:02.360463 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:58:02.365 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:02.360779 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:58:03.963 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:03.962914 I | op-mon: targeting the mon count 3 2022-03-21T14:58:03.972 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:03.972064 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:58:04.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:04.331968 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:58:04.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:04.332009 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:58:04.692 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:04.691132 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:58:04.692 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:04.691168 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:58:05.038 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:05.038060 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:58:05.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:05.038084 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:58:05.396 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:05.395560 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:58:05.396 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:05.395589 I | op-config: deleting "log file" option from the mon configuration database 2022-03-21T14:58:05.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:05.753879 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-21T14:58:05.755 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:05.753911 I | op-mon: checking for basic quorum with existing mons 2022-03-21T14:58:05.957 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:05.955265 I | op-mon: mon "a" ip is 10.97.100.133 2022-03-21T14:58:06.133 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:06.133123 I | op-mon: mon "b" ip is 10.96.127.40 2022-03-21T14:58:06.364 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:06.364127 I | op-mon: mon "c" ip is 10.111.185.43 2022-03-21T14:58:06.963 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:06.962285 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789","10.111.185.43:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:58:07.562 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:07.561555 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:58:07.567 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:07.561896 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:58:07.985 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:07.984306 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-21T14:58:07.993 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:07.993144 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-21T14:58:07.993 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:07.993173 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:08.572 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:08.572071 I | op-mon: mons running: [a b c] 2022-03-21T14:58:09.011 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.010077 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:09.018 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.017632 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-21T14:58:09.030 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.030431 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-21T14:58:09.031 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.030455 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:09.172 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.171302 I | op-mon: mons running: [a b c] 2022-03-21T14:58:09.593 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.592405 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:09.600 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.599747 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-21T14:58:09.609 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.608262 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-21T14:58:09.609 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.608282 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:09.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:09.771299 I | op-mon: mons running: [a b c] 2022-03-21T14:58:10.190 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:10.189158 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:10.190 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:10.189192 I | op-mon: mons created: 3 2022-03-21T14:58:10.597 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:10.597054 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:10.635 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:10.634788 I | op-mon: mons running: [a b c] 2022-03-21T14:58:11.045 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:11.044069 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:11.045 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:11.044149 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-21T14:58:11.045 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:11.044229 I | op-mon: start running mons 2022-03-21T14:58:11.054 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:11.052176 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:58:11.503 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:11.502532 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-21T14:58:11.763 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:11.762565 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789","10.111.185.43:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:58:11.935 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:11.935160 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-21T14:58:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:+ '[' 11 '!=' 12 ']' 2022-03-21T14:58:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:+ echo 'waiting for osd to come back up' 2022-03-21T14:58:12.112 INFO:teuthology.orchestra.run.smithi043.stdout:waiting for osd to come back up 2022-03-21T14:58:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:++ toolbox ceph osd stat 2022-03-21T14:58:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-21T14:58:12.113 INFO:teuthology.orchestra.run.smithi043.stderr:++ cut -f3 -d ' ' 2022-03-21T14:58:12.189 INFO:teuthology.orchestra.run.smithi043.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-21T14:58:12.360 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:12.359459 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:58:12.365 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:12.359698 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:58:12.370 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:12.369756 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-21T14:58:12.714 INFO:teuthology.orchestra.run.smithi043.stderr:+ num_osd=11 2022-03-21T14:58:12.714 INFO:teuthology.orchestra.run.smithi043.stderr:+ sleep 30s 2022-03-21T14:58:14.964 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:14.962903 I | op-mon: targeting the mon count 3 2022-03-21T14:58:14.971 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:14.971115 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:58:15.345 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:15.344271 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:58:15.346 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:15.344313 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:58:15.564 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:15.563065 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-21T14:58:15.564 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:15.563099 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-21T14:58:15.697 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:15.696820 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:58:15.698 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:15.696844 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:58:16.075 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.074370 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:58:16.076 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.074411 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:58:16.083 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.083005 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-21T14:58:16.084 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.083026 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-21T14:58:16.084 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.083030 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-21T14:58:16.547 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.545788 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:58:16.547 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.545826 I | op-config: deleting "log file" option from the mon configuration database 2022-03-21T14:58:16.661 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.659576 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-21T14:58:16.917 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.728867 I | op-mgr: start running mgr 2022-03-21T14:58:16.917 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.728906 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-21T14:58:16.956 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.955186 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-21T14:58:16.957 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:16.955221 I | op-mon: checking for basic quorum with existing mons 2022-03-21T14:58:17.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:17.230654 I | op-mon: mon "a" ip is 10.97.100.133 2022-03-21T14:58:18.293 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:18.292704 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-21T14:58:18.305 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:18.305122 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-21T14:58:18.364 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:18.363592 I | op-mgr: setting services to point to mgr "a" 2022-03-21T14:58:18.568 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:18.567082 I | op-mon: mon "b" ip is 10.96.127.40 2022-03-21T14:58:20.167 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.165788 I | op-mon: mon "c" ip is 10.111.185.43 2022-03-21T14:58:20.361 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.360319 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-21T14:58:20.361 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.360395 I | op-mgr: successful modules: balancer 2022-03-21T14:58:20.399 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.398690 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-21T14:58:20.400 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.398726 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-21T14:58:20.445 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.444840 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-21T14:58:20.767 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764155 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-1" 2022-03-21T14:58:20.768 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764208 I | op-osd: OSD PVC "smithi043-lv-1-data-0vfptn" already exists 2022-03-21T14:58:20.768 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764230 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-2" 2022-03-21T14:58:20.768 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764256 I | op-osd: OSD PVC "smithi043-lv-2-data-0jwsgt" already exists 2022-03-21T14:58:20.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764278 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-3" 2022-03-21T14:58:20.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764301 I | op-osd: OSD PVC "smithi043-lv-3-data-0p8lrq" already exists 2022-03-21T14:58:20.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764320 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-4" 2022-03-21T14:58:20.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764346 I | op-osd: OSD PVC "smithi043-lv-4-data-0zg56f" already exists 2022-03-21T14:58:20.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764365 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-1" 2022-03-21T14:58:20.770 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764388 I | op-osd: OSD PVC "smithi065-lv-1-data-0zrnzn" already exists 2022-03-21T14:58:20.770 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764410 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-2" 2022-03-21T14:58:20.770 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764433 I | op-osd: OSD PVC "smithi065-lv-2-data-0fgs82" already exists 2022-03-21T14:58:20.770 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764461 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-3" 2022-03-21T14:58:20.771 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764494 I | op-osd: OSD PVC "smithi065-lv-3-data-0pnlwn" already exists 2022-03-21T14:58:20.771 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764518 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-4" 2022-03-21T14:58:20.771 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764547 I | op-osd: OSD PVC "smithi065-lv-4-data-0jcxd2" already exists 2022-03-21T14:58:20.771 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764568 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-2" 2022-03-21T14:58:20.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764597 I | op-osd: OSD PVC "smithi154-lv-2-data-0n64sd" already exists 2022-03-21T14:58:20.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764632 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-3" 2022-03-21T14:58:20.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764664 I | op-osd: OSD PVC "smithi154-lv-3-data-0vf74c" already exists 2022-03-21T14:58:20.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764687 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-4" 2022-03-21T14:58:20.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764716 I | op-osd: OSD PVC "smithi154-lv-4-data-0h7xg6" already exists 2022-03-21T14:58:20.773 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:20.764738 I | op-osd: creating 1 new PVCs for device set "smithi154-lv-1" 2022-03-21T14:58:21.165 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.164581 I | op-osd: successfully provisioned PVC "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:58:21.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195405 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-1-data-0vfptn" 2022-03-21T14:58:21.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195432 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-1-data-0vfptn" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195439 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:58:21.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195444 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-2-data-0jwsgt" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195449 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:58:21.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195454 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-3-data-0p8lrq" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195459 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-4-data-0zg56f" 2022-03-21T14:58:21.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195463 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-4-data-0zg56f" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195468 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:58:21.201 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195473 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-1-data-0zrnzn" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.201 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195478 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-2-data-0fgs82" 2022-03-21T14:58:21.201 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195482 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-2-data-0fgs82" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.201 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195489 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:58:21.202 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195495 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-3-data-0pnlwn" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.202 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195500 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:58:21.202 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195504 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-4-data-0jcxd2" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.202 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195510 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-2-data-0n64sd" 2022-03-21T14:58:21.202 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195515 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-2-data-0n64sd" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.203 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195520 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-3-data-0vf74c" 2022-03-21T14:58:21.203 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195525 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-3-data-0vf74c" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.203 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195530 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:58:21.203 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195535 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-4-data-0h7xg6" because OSD daemon using the PVC already exists 2022-03-21T14:58:21.204 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.195541 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:58:21.363 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.362114 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789","10.111.185.43:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:58:21.928 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.926380 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-21T14:58:21.929 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.926460 I | op-mgr: successful modules: dashboard 2022-03-21T14:58:21.930 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.927809 I | op-mgr: successful modules: prometheus 2022-03-21T14:58:21.977 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.976444 I | op-osd: started OSD provisioning job for PVC "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:58:21.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.976468 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-21T14:58:21.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:21.976473 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-21T14:58:22.361 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.360430 I | op-osd: OSD orchestration status for PVC smithi154-lv-1-data-0r7ptm is "starting" 2022-03-21T14:58:22.463 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.462183 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:22.469 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.468334 I | op-osd: updating OSD 1 on PVC "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:58:22.469 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.468370 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:58:22.572 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.571674 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:58:22.573 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.571967 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:58:22.650 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.648832 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:22.656 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.655702 I | op-osd: updating OSD 10 on PVC "smithi043-lv-4-data-0zg56f" 2022-03-21T14:58:22.657 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.655738 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-4-data-0zg56f" 2022-03-21T14:58:22.828 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.826704 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:22.833 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.831472 I | op-osd: updating OSD 11 on PVC "smithi043-lv-1-data-0vfptn" 2022-03-21T14:58:22.833 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:22.831491 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-1-data-0vfptn" 2022-03-21T14:58:23.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:23.197710 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:23.357 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:23.355964 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-21T14:58:23.473 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:23.472575 I | op-osd: updating OSD 2 on PVC "smithi065-lv-2-data-0fgs82" 2022-03-21T14:58:23.474 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:23.472615 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-2-data-0fgs82" 2022-03-21T14:58:23.552 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:23.551169 I | op-mgr: successful modules: orchestrator modules 2022-03-21T14:58:23.678 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:23.676824 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-21T14:58:23.678 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:23.676849 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:23.726 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:23.725496 I | op-mon: mons running: [a b c] 2022-03-21T14:58:24.137 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:24.136566 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:24.375 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:24.374757 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:24.472 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:24.471651 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-21T14:58:24.723 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:24.722076 I | op-osd: updating OSD 3 on PVC "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:58:24.723 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:24.722113 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:58:24.878 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:24.877111 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-21T14:58:24.878 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:24.877149 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:24.917 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:24.916425 I | op-mon: mons running: [a b c] 2022-03-21T14:58:25.345 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:25.343997 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:25.578 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:25.577414 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:25.671 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:25.669932 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-21T14:58:25.870 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:25.868783 I | op-osd: updating OSD 4 on PVC "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:58:25.870 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:25.868820 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:58:26.078 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:26.077216 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-21T14:58:26.079 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:26.077258 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:26.145 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:26.144610 I | op-mon: mons running: [a b c] 2022-03-21T14:58:26.566 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:26.565239 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:26.567 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:26.565271 I | op-mon: mons created: 3 2022-03-21T14:58:26.775 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:26.774363 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:26.872 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:26.870830 I | op-osd: updating OSD 5 on PVC "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:58:26.872 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:26.870872 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:58:26.986 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:26.984662 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:27.021 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:27.020710 I | op-mon: mons running: [a b c] 2022-03-21T14:58:27.448 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:27.446662 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:27.448 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:27.446715 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-21T14:58:27.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:27.574229 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:27.670 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:27.669604 I | op-osd: updating OSD 6 on PVC "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:58:27.671 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:27.669638 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:58:27.883 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:27.882019 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-21T14:58:28.324 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:28.323281 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-21T14:58:28.378 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:28.376868 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:28.486 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:28.484854 I | op-osd: updating OSD 7 on PVC "smithi154-lv-3-data-0vf74c" 2022-03-21T14:58:28.486 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:28.484891 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-3-data-0vf74c" 2022-03-21T14:58:29.025 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.023877 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-21T14:58:29.219 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.218262 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:29.271 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.269813 I | op-osd: updating OSD 8 on PVC "smithi154-lv-2-data-0n64sd" 2022-03-21T14:58:29.271 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.269857 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-2-data-0n64sd" 2022-03-21T14:58:29.483 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.482624 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-21T14:58:29.484 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.482642 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-21T14:58:29.949 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.948414 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-21T14:58:29.950 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.948450 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-21T14:58:29.950 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.948459 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-21T14:58:29.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:29.977046 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:30.069 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:30.068790 I | op-osd: updating OSD 9 on PVC "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:58:30.070 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:30.068811 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:58:30.377 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:30.375699 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-21T14:58:30.405 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:30.404861 I | op-mgr: start running mgr 2022-03-21T14:58:30.406 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:30.404902 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-21T14:58:30.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:30.900310 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-21T14:58:31.034 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.033350 I | op-osd: OSD orchestration status for node smithi154-lv-1-data-0r7ptm is "orchestrating" 2022-03-21T14:58:31.075 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.074122 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-21T14:58:31.274 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.269821 I | op-mgr: setting services to point to mgr "a" 2022-03-21T14:58:31.428 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.427170 I | op-osd: OSD orchestration status for PVC smithi154-lv-1-data-0r7ptm is "orchestrating" 2022-03-21T14:58:31.459 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.457571 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-21T14:58:31.471 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.457671 I | op-mgr: successful modules: balancer 2022-03-21T14:58:31.484 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.483230 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-21T14:58:31.484 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.483268 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-21T14:58:31.570 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.569277 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-21T14:58:31.724 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720693 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-1" 2022-03-21T14:58:31.724 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720732 I | op-osd: OSD PVC "smithi043-lv-1-data-0vfptn" already exists 2022-03-21T14:58:31.724 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720745 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-2" 2022-03-21T14:58:31.724 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720756 I | op-osd: OSD PVC "smithi043-lv-2-data-0jwsgt" already exists 2022-03-21T14:58:31.725 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720763 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-3" 2022-03-21T14:58:31.725 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720773 I | op-osd: OSD PVC "smithi043-lv-3-data-0p8lrq" already exists 2022-03-21T14:58:31.725 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720780 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-4" 2022-03-21T14:58:31.725 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720789 I | op-osd: OSD PVC "smithi043-lv-4-data-0zg56f" already exists 2022-03-21T14:58:31.725 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720795 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-1" 2022-03-21T14:58:31.726 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720803 I | op-osd: OSD PVC "smithi065-lv-1-data-0zrnzn" already exists 2022-03-21T14:58:31.726 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720811 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-2" 2022-03-21T14:58:31.726 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720820 I | op-osd: OSD PVC "smithi065-lv-2-data-0fgs82" already exists 2022-03-21T14:58:31.726 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720826 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-3" 2022-03-21T14:58:31.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720835 I | op-osd: OSD PVC "smithi065-lv-3-data-0pnlwn" already exists 2022-03-21T14:58:31.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720841 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-4" 2022-03-21T14:58:31.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720849 I | op-osd: OSD PVC "smithi065-lv-4-data-0jcxd2" already exists 2022-03-21T14:58:31.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720855 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-2" 2022-03-21T14:58:31.728 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720866 I | op-osd: OSD PVC "smithi154-lv-2-data-0n64sd" already exists 2022-03-21T14:58:31.728 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720876 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-3" 2022-03-21T14:58:31.728 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720885 I | op-osd: OSD PVC "smithi154-lv-3-data-0vf74c" already exists 2022-03-21T14:58:31.728 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720892 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-4" 2022-03-21T14:58:31.728 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720902 I | op-osd: OSD PVC "smithi154-lv-4-data-0h7xg6" already exists 2022-03-21T14:58:31.729 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720908 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-1" 2022-03-21T14:58:31.729 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.720915 I | op-osd: OSD PVC "smithi154-lv-1-data-0r7ptm" already exists 2022-03-21T14:58:31.750 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747434 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-1-data-0vfptn" 2022-03-21T14:58:31.750 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747458 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-1-data-0vfptn" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.751 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747464 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:58:31.751 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747469 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-2-data-0jwsgt" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.751 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747474 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:58:31.751 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747479 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-3-data-0p8lrq" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.751 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747484 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-4-data-0zg56f" 2022-03-21T14:58:31.752 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747489 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-4-data-0zg56f" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.752 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747493 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:58:31.752 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747498 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-1-data-0zrnzn" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.752 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747502 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-2-data-0fgs82" 2022-03-21T14:58:31.753 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747507 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-2-data-0fgs82" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.753 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747512 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:58:31.753 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747517 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-3-data-0pnlwn" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.753 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747522 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:58:31.753 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747527 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-4-data-0jcxd2" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747532 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-2-data-0n64sd" 2022-03-21T14:58:31.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747537 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-2-data-0n64sd" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747543 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-3-data-0vf74c" 2022-03-21T14:58:31.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747548 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-3-data-0vf74c" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.755 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747553 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:58:31.755 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747558 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-4-data-0h7xg6" because OSD daemon using the PVC already exists 2022-03-21T14:58:31.755 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:31.747563 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:58:32.062 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.060794 I | op-osd: OSD orchestration status for PVC smithi154-lv-1-data-0r7ptm is "starting" 2022-03-21T14:58:32.073 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.072592 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi154-lv-1-data-0r7ptm. Status={Conditions:[] StartTime:2022-03-21 14:58:21 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-21T14:58:32.074 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.072613 I | op-osd: started OSD provisioning job for PVC "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:58:32.074 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.072621 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-21T14:58:32.074 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.072626 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-21T14:58:32.258 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.256871 I | op-osd: OSD orchestration status for PVC smithi154-lv-1-data-0r7ptm is "starting" 2022-03-21T14:58:32.360 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.358740 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:32.397 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.396193 I | op-osd: updating OSD 1 on PVC "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:58:32.397 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.396224 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:58:32.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.574631 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:32.674 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.673689 I | op-osd: updating OSD 10 on PVC "smithi043-lv-4-data-0zg56f" 2022-03-21T14:58:32.675 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.673723 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-4-data-0zg56f" 2022-03-21T14:58:32.968 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.966478 I | op-mgr: successful modules: prometheus 2022-03-21T14:58:32.968 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.966601 I | op-mgr: successful modules: dashboard 2022-03-21T14:58:32.968 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:32.966820 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-21T14:58:33.377 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:33.376455 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:33.471 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:33.470157 I | op-osd: updating OSD 11 on PVC "smithi043-lv-1-data-0vfptn" 2022-03-21T14:58:33.471 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:33.470193 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-1-data-0vfptn" 2022-03-21T14:58:34.176 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:34.174979 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:34.271 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:34.270637 I | op-osd: updating OSD 2 on PVC "smithi065-lv-2-data-0fgs82" 2022-03-21T14:58:34.272 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:34.270676 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-2-data-0fgs82" 2022-03-21T14:58:34.628 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:34.627119 I | op-mgr: successful modules: orchestrator modules 2022-03-21T14:58:34.977 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:34.976326 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:35.071 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:35.070684 I | op-osd: updating OSD 3 on PVC "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:58:35.072 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:35.070720 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:58:35.777 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:35.776236 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:35.870 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:35.869460 I | op-osd: updating OSD 4 on PVC "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:58:35.871 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:35.869502 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:58:36.578 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:36.577136 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:36.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:36.686430 I | op-osd: updating OSD 5 on PVC "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:58:36.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:36.686476 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:58:37.378 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:37.377107 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:37.440 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:37.438947 I | op-osd: OSD orchestration status for PVC smithi154-lv-1-data-0r7ptm is "completed" 2022-03-21T14:58:37.440 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:37.438981 I | op-osd: creating OSD 0 on PVC "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:58:37.440 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:37.438995 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:58:37.470 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:37.468911 I | op-osd: updating OSD 6 on PVC "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:58:37.470 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:37.468948 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:58:38.490 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:38.489542 I | op-osd: OSD orchestration status for PVC smithi154-lv-1-data-0r7ptm is "completed" 2022-03-21T14:58:38.491 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:38.489581 I | op-osd: creating OSD 0 on PVC "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:58:38.491 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:38.489603 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:58:38.598 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:38.597391 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-21T14:58:38.599 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:38.597417 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:58:38.916 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:38.915680 E | op-osd: failed to create OSD 0 on PVC "smithi154-lv-1-data-0r7ptm": failed to create deployment for OSD 0 on PVC "smithi154-lv-1-data-0r7ptm": deployments.apps "rook-ceph-osd-0" already exists 2022-03-21T14:58:39.020 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:39.018749 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:39.071 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:39.069655 I | op-osd: updating OSD 7 on PVC "smithi154-lv-3-data-0vf74c" 2022-03-21T14:58:39.071 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:39.069681 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-3-data-0vf74c" 2022-03-21T14:58:39.785 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:39.784193 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:39.870 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:39.869345 I | op-osd: updating OSD 8 on PVC "smithi154-lv-2-data-0n64sd" 2022-03-21T14:58:39.871 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:39.869374 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-2-data-0n64sd" 2022-03-21T14:58:40.578 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:40.576827 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:58:40.687 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:40.685764 I | op-osd: updating OSD 9 on PVC "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:58:40.687 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:40.685803 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:58:41.310 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:41.308250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-21T14:58:41.310 INFO:tasks.rook.operator.smithi043.stdout:failed to create OSD 0 on PVC "smithi154-lv-1-data-0r7ptm": failed to create deployment for OSD 0 on PVC "smithi154-lv-1-data-0r7ptm": deployments.apps "rook-ceph-osd-0" already exists 2022-03-21T14:58:41.315 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:41.314422 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:58:41.348 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:41.346883 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:58:41.485 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:41.484310 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-21T14:58:42.716 INFO:teuthology.orchestra.run.smithi043.stderr:+ '[' 11 '!=' 12 ']' 2022-03-21T14:58:42.716 INFO:teuthology.orchestra.run.smithi043.stdout:waiting for osd to come back up 2022-03-21T14:58:42.717 INFO:teuthology.orchestra.run.smithi043.stderr:+ echo 'waiting for osd to come back up' 2022-03-21T14:58:42.717 INFO:teuthology.orchestra.run.smithi043.stderr:++ toolbox ceph osd stat 2022-03-21T14:58:42.718 INFO:teuthology.orchestra.run.smithi043.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-21T14:58:42.718 INFO:teuthology.orchestra.run.smithi043.stderr:++ cut -f3 -d ' ' 2022-03-21T14:58:42.804 INFO:teuthology.orchestra.run.smithi043.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-21T14:58:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:+ num_osd=11 2022-03-21T14:58:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:+ sleep 30s 2022-03-21T14:58:48.951 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:48.949703 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-21T14:58:48.951 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:48.949732 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-21T14:58:48.958 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:48.956760 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:58:49.002 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:49.001228 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:58:49.003 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:49.001530 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:58:49.428 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:49.427308 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-21T14:58:49.538 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:49.536793 I | op-mon: start running mons 2022-03-21T14:58:49.568 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:49.567348 I | op-mon: parsing mon endpoints: a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 2022-03-21T14:58:49.609 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:49.607962 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.96.127.40:6789","10.111.185.43:6789","10.97.100.133:6789"]}] data:a=10.97.100.133:6789,b=10.96.127.40:6789,c=10.111.185.43:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:58:50.011 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:50.010021 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:58:50.015 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:50.010342 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:58:51.613 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:51.611876 I | op-mon: targeting the mon count 3 2022-03-21T14:58:51.625 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:51.624381 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:58:52.157 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:52.155769 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-21T14:58:52.157 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:52.155807 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:58:52.770 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:52.768411 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-21T14:58:52.770 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:52.768449 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:58:53.132 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:53.130239 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-21T14:58:53.132 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:53.130271 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:58:53.583 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:53.582034 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-21T14:58:53.584 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:53.582066 I | op-config: deleting "log file" option from the mon configuration database 2022-03-21T14:58:53.927 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:53.925704 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-21T14:58:53.927 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:53.925724 I | op-mon: checking for basic quorum with existing mons 2022-03-21T14:58:54.034 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:54.032623 I | op-mon: mon "a" ip is 10.97.100.133 2022-03-21T14:58:54.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:54.194218 I | op-mon: mon "b" ip is 10.96.127.40 2022-03-21T14:58:54.410 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:54.408855 I | op-mon: mon "c" ip is 10.111.185.43 2022-03-21T14:58:54.931 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:54.930297 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.100.133:6789","10.96.127.40:6789","10.111.185.43:6789"]}] data:b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 mapping:{"node":{"a":{"Name":"smithi154","Hostname":"smithi154","Address":"172.21.15.154"},"b":{"Name":"smithi065","Hostname":"smithi065","Address":"172.21.15.65"},"c":{"Name":"smithi043","Hostname":"smithi043","Address":"172.21.15.43"}}} maxMonId:2] 2022-03-21T14:58:55.531 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:55.529766 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-21T14:58:55.532 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:55.530046 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-21T14:58:55.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:55.976477 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-21T14:58:55.986 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:55.985350 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-21T14:58:55.987 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:55.985374 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:56.543 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:56.541867 I | op-mon: mons running: [a b c] 2022-03-21T14:58:56.955 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:56.953330 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:56.963 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:56.961863 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-21T14:58:56.972 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:56.971693 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-21T14:58:56.973 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:56.971724 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:57.142 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:57.141069 I | op-mon: mons running: [a b c] 2022-03-21T14:58:57.588 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:57.587234 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:57.596 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:57.595033 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-21T14:58:57.606 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:57.605302 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-21T14:58:57.606 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:57.605320 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:57.743 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:57.741939 I | op-mon: mons running: [a b c] 2022-03-21T14:58:58.206 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:58.204423 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:58.206 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:58.204454 I | op-mon: mons created: 3 2022-03-21T14:58:58.651 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:58.649910 I | op-mon: waiting for mon quorum with [a b c] 2022-03-21T14:58:58.695 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:58.693733 I | op-mon: mons running: [a b c] 2022-03-21T14:58:59.107 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:59.106219 I | op-mon: Monitors in quorum: [a b c] 2022-03-21T14:58:59.108 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:59.106284 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-21T14:58:59.663 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:58:59.661465 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-21T14:59:00.097 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:00.096167 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-21T14:59:00.515 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:00.514133 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-21T14:59:01.016 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:01.015026 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-21T14:59:01.017 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:01.015057 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-21T14:59:01.447 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:01.445595 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-21T14:59:01.447 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:01.445623 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-21T14:59:01.447 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:01.445630 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-21T14:59:01.909 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:01.907916 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-21T14:59:01.951 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:01.946122 I | op-mgr: start running mgr 2022-03-21T14:59:01.952 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:01.946175 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-21T14:59:02.468 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:02.466542 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-21T14:59:02.477 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:02.475785 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-21T14:59:02.488 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:02.487080 I | op-mgr: setting services to point to mgr "a" 2022-03-21T14:59:02.993 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:02.991272 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-21T14:59:02.993 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:02.991386 I | op-mgr: successful modules: balancer 2022-03-21T14:59:03.004 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.003237 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-21T14:59:03.005 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.003274 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-21T14:59:03.064 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.062729 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-21T14:59:03.194 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192548 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-1" 2022-03-21T14:59:03.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192593 I | op-osd: OSD PVC "smithi043-lv-1-data-0vfptn" already exists 2022-03-21T14:59:03.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192614 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-2" 2022-03-21T14:59:03.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192634 I | op-osd: OSD PVC "smithi043-lv-2-data-0jwsgt" already exists 2022-03-21T14:59:03.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192652 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-3" 2022-03-21T14:59:03.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192673 I | op-osd: OSD PVC "smithi043-lv-3-data-0p8lrq" already exists 2022-03-21T14:59:03.196 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192690 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi043-lv-4" 2022-03-21T14:59:03.197 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192709 I | op-osd: OSD PVC "smithi043-lv-4-data-0zg56f" already exists 2022-03-21T14:59:03.197 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192724 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-1" 2022-03-21T14:59:03.197 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192743 I | op-osd: OSD PVC "smithi065-lv-1-data-0zrnzn" already exists 2022-03-21T14:59:03.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192759 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-2" 2022-03-21T14:59:03.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192778 I | op-osd: OSD PVC "smithi065-lv-2-data-0fgs82" already exists 2022-03-21T14:59:03.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192795 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-3" 2022-03-21T14:59:03.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192813 I | op-osd: OSD PVC "smithi065-lv-3-data-0pnlwn" already exists 2022-03-21T14:59:03.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192826 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi065-lv-4" 2022-03-21T14:59:03.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192843 I | op-osd: OSD PVC "smithi065-lv-4-data-0jcxd2" already exists 2022-03-21T14:59:03.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192856 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-2" 2022-03-21T14:59:03.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192875 I | op-osd: OSD PVC "smithi154-lv-2-data-0n64sd" already exists 2022-03-21T14:59:03.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192898 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-3" 2022-03-21T14:59:03.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192919 I | op-osd: OSD PVC "smithi154-lv-3-data-0vf74c" already exists 2022-03-21T14:59:03.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192937 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-4" 2022-03-21T14:59:03.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192948 I | op-osd: OSD PVC "smithi154-lv-4-data-0h7xg6" already exists 2022-03-21T14:59:03.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192954 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi154-lv-1" 2022-03-21T14:59:03.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.192966 I | op-osd: OSD PVC "smithi154-lv-1-data-0r7ptm" already exists 2022-03-21T14:59:03.227 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223921 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-1-data-0vfptn" 2022-03-21T14:59:03.228 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223952 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-1-data-0vfptn" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.228 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223959 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:59:03.228 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223964 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-2-data-0jwsgt" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.228 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223969 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:59:03.228 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223974 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-3-data-0p8lrq" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223979 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-4-data-0zg56f" 2022-03-21T14:59:03.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223984 I | op-osd: skipping OSD prepare job creation for PVC "smithi043-lv-4-data-0zg56f" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223989 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:59:03.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223994 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-1-data-0zrnzn" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.223999 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-2-data-0fgs82" 2022-03-21T14:59:03.230 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224004 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-2-data-0fgs82" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.230 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224010 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:59:03.230 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224014 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-3-data-0pnlwn" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.230 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224020 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:59:03.230 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224025 I | op-osd: skipping OSD prepare job creation for PVC "smithi065-lv-4-data-0jcxd2" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224030 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-2-data-0n64sd" 2022-03-21T14:59:03.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224035 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-2-data-0n64sd" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224040 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-3-data-0vf74c" 2022-03-21T14:59:03.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224045 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-3-data-0vf74c" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224051 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:59:03.232 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224067 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-4-data-0h7xg6" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.232 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224072 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:59:03.232 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224077 I | op-osd: skipping OSD prepare job creation for PVC "smithi154-lv-1-data-0r7ptm" because OSD daemon using the PVC already exists 2022-03-21T14:59:03.232 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224111 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-21T14:59:03.233 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.224120 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-21T14:59:03.494 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.492607 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:03.503 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.501963 I | op-osd: updating OSD 0 on PVC "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:59:03.504 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.502136 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-1-data-0r7ptm" 2022-03-21T14:59:03.971 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.969811 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:03.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.976732 I | op-osd: updating OSD 1 on PVC "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:59:03.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:03.976770 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-3-data-0pnlwn" 2022-03-21T14:59:04.156 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.155138 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:04.163 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.161949 I | op-osd: updating OSD 10 on PVC "smithi043-lv-4-data-0zg56f" 2022-03-21T14:59:04.163 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.161984 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-4-data-0zg56f" 2022-03-21T14:59:04.350 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.349166 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-21T14:59:04.354 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.353297 I | op-mgr: successful modules: dashboard 2022-03-21T14:59:04.367 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.365473 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:04.371 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.369586 I | op-mgr: successful modules: prometheus 2022-03-21T14:59:04.379 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.377304 I | op-osd: updating OSD 11 on PVC "smithi043-lv-1-data-0vfptn" 2022-03-21T14:59:04.379 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.377333 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-1-data-0vfptn" 2022-03-21T14:59:04.823 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.821818 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:04.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.899930 I | op-osd: updating OSD 2 on PVC "smithi065-lv-2-data-0fgs82" 2022-03-21T14:59:04.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:04.899967 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-2-data-0fgs82" 2022-03-21T14:59:05.605 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:05.603406 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:05.701 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:05.699627 I | op-osd: updating OSD 3 on PVC "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:59:05.701 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:05.699667 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-3-data-0p8lrq" 2022-03-21T14:59:05.791 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:05.790266 I | op-mgr: successful modules: orchestrator modules 2022-03-21T14:59:06.414 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:06.412383 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:06.502 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:06.500585 I | op-osd: updating OSD 4 on PVC "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:59:06.502 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:06.500621 I | op-osd: OSD will have its main bluestore block on "smithi043-lv-2-data-0jwsgt" 2022-03-21T14:59:07.208 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:07.207019 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:07.300 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:07.298747 I | op-osd: updating OSD 5 on PVC "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:59:07.300 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:07.298784 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-1-data-0zrnzn" 2022-03-21T14:59:08.009 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:08.007348 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:08.102 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:08.100880 I | op-osd: updating OSD 6 on PVC "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:59:08.103 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:08.100920 I | op-osd: OSD will have its main bluestore block on "smithi065-lv-4-data-0jcxd2" 2022-03-21T14:59:08.805 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:08.803596 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:08.900 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:08.899112 I | op-osd: updating OSD 7 on PVC "smithi154-lv-3-data-0vf74c" 2022-03-21T14:59:08.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:08.899136 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-3-data-0vf74c" 2022-03-21T14:59:09.609 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:09.608029 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:09.701 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:09.699845 I | op-osd: updating OSD 8 on PVC "smithi154-lv-2-data-0n64sd" 2022-03-21T14:59:09.701 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:09.699884 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-2-data-0n64sd" 2022-03-21T14:59:10.409 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:10.408223 I | op-osd: skipping osd checks for ok-to-stop 2022-03-21T14:59:10.542 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:10.541158 I | op-osd: updating OSD 9 on PVC "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:59:10.543 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:10.541198 I | op-osd: OSD will have its main bluestore block on "smithi154-lv-4-data-0h7xg6" 2022-03-21T14:59:11.598 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:11.596299 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-21T14:59:11.598 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:11.596335 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-21T14:59:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:+ '[' 11 '!=' 12 ']' 2022-03-21T14:59:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:+ echo 'waiting for osd to come back up' 2022-03-21T14:59:13.292 INFO:teuthology.orchestra.run.smithi043.stdout:waiting for osd to come back up 2022-03-21T14:59:13.293 INFO:teuthology.orchestra.run.smithi043.stderr:++ toolbox ceph osd stat 2022-03-21T14:59:13.293 INFO:teuthology.orchestra.run.smithi043.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-21T14:59:13.294 INFO:teuthology.orchestra.run.smithi043.stderr:++ cut -f3 -d ' ' 2022-03-21T14:59:13.368 INFO:teuthology.orchestra.run.smithi043.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-21T14:59:13.848 INFO:teuthology.orchestra.run.smithi043.stderr:+ num_osd=12 2022-03-21T14:59:13.849 INFO:teuthology.orchestra.run.smithi043.stderr:+ sleep 30s 2022-03-21T14:59:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:+ '[' 12 '!=' 12 ']' 2022-03-21T14:59:43.852 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-21T14:59:43.863 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch status 2022-03-21T14:59:44.586 INFO:teuthology.orchestra.run.smithi043.stdout:Backend: rook 2022-03-21T14:59:44.587 INFO:teuthology.orchestra.run.smithi043.stdout:Available: Yes 2022-03-21T14:59:44.610 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ps 2022-03-21T14:59:45.233 INFO:teuthology.orchestra.run.smithi043.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-21T14:59:45.233 INFO:teuthology.orchestra.run.smithi043.stdout:crashcollector.crash smithi043 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.234 INFO:teuthology.orchestra.run.smithi043.stdout:crashcollector.crash smithi065 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-21T14:59:45.234 INFO:teuthology.orchestra.run.smithi043.stdout:crashcollector.crash smithi154 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-21T14:59:45.234 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.a smithi043 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-21T14:59:45.234 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a smithi154 running (7m) 0s ago 7m - - 7c0c85ada2e4 2022-03-21T14:59:45.235 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b smithi065 running (6m) 0s ago 6m - - 7c0c85ada2e4 2022-03-21T14:59:45.235 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c smithi043 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-21T14:59:45.235 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 smithi154 running (67s) 0s ago 67s - - 7c0c85ada2e4 2022-03-21T14:59:45.235 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 smithi065 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.235 INFO:teuthology.orchestra.run.smithi043.stdout:osd.10 smithi043 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.236 INFO:teuthology.orchestra.run.smithi043.stdout:osd.11 smithi043 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.236 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 smithi065 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.236 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 smithi043 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.236 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 smithi043 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.236 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 smithi065 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.237 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 smithi065 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.237 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 smithi154 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.237 INFO:teuthology.orchestra.run.smithi043.stdout:osd.8 smithi154 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.237 INFO:teuthology.orchestra.run.smithi043.stdout:osd.9 smithi154 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-21T14:59:45.256 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls 2022-03-21T14:59:45.962 INFO:teuthology.orchestra.run.smithi043.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-21T14:59:45.963 INFO:teuthology.orchestra.run.smithi043.stdout:crash 3/3 0s ago 4m * 2022-03-21T14:59:45.963 INFO:teuthology.orchestra.run.smithi043.stdout:mgr 1/1 0s ago 5m count:1 2022-03-21T14:59:45.963 INFO:teuthology.orchestra.run.smithi043.stdout:mon 3/3 0s ago 7m count:3 2022-03-21T14:59:45.963 INFO:teuthology.orchestra.run.smithi043.stdout:osd 24 0s ago 3m 2022-03-21T14:59:45.964 INFO:teuthology.orchestra.run.smithi043.stdout:osd.all-available-devices 0 0s ago - * 2022-03-21T14:59:45.989 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch host ls 2022-03-21T14:59:46.547 INFO:teuthology.orchestra.run.smithi043.stdout:HOST ADDR LABELS STATUS 2022-03-21T14:59:46.547 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 172.21.15.43/smithi043 2022-03-21T14:59:46.547 INFO:teuthology.orchestra.run.smithi043.stdout:smithi065 172.21.15.65/smithi065 2022-03-21T14:59:46.548 INFO:teuthology.orchestra.run.smithi043.stdout:smithi154 172.21.15.154/smithi154 2022-03-21T14:59:46.548 INFO:teuthology.orchestra.run.smithi043.stdout:3 hosts in cluster 2022-03-21T14:59:46.567 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch device ls 2022-03-21T14:59:47.193 INFO:teuthology.orchestra.run.smithi043.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-21T14:59:47.193 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-21T14:59:47.193 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-21T14:59:47.193 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-21T14:59:47.194 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-21T14:59:47.194 INFO:teuthology.orchestra.run.smithi043.stdout:smithi065 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-21T14:59:47.194 INFO:teuthology.orchestra.run.smithi043.stdout:smithi065 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-21T14:59:47.194 INFO:teuthology.orchestra.run.smithi043.stdout:smithi065 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-21T14:59:47.195 INFO:teuthology.orchestra.run.smithi043.stdout:smithi065 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-21T14:59:47.195 INFO:teuthology.orchestra.run.smithi043.stdout:smithi154 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-21T14:59:47.195 INFO:teuthology.orchestra.run.smithi043.stdout:smithi154 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-21T14:59:47.195 INFO:teuthology.orchestra.run.smithi043.stdout:smithi154 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-21T14:59:47.196 INFO:teuthology.orchestra.run.smithi043.stdout:smithi154 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-21T14:59:47.212 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch apply rgw foo 2022-03-21T14:59:47.753 INFO:teuthology.orchestra.run.smithi043.stdout:Created 2022-03-21T14:59:47.762 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:47.757358 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-21T14:59:47.779 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch apply mds foo 2022-03-21T14:59:47.860 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:47.858425 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T14:59:47.860 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:47.858514 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-21T14:59:48.326 INFO:teuthology.orchestra.run.smithi043.stdout:Created 2022-03-21T14:59:48.328 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:48.327496 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-21T14:59:48.349 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch apply rbd-mirror 2022-03-21T14:59:48.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:48.392656 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T14:59:48.398 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:48.397094 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T14:59:48.398 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:48.397129 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-21T14:59:49.411 INFO:teuthology.orchestra.run.smithi043.stdout:Success 2022-03-21T14:59:49.431 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:49.429939 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T14:59:49.431 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:49.429984 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-21T14:59:49.435 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch apply nfs foo --port 12777 2022-03-21T14:59:53.587 INFO:teuthology.orchestra.run.smithi043.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-21T14:59:53.587 INFO:teuthology.orchestra.run.smithi043.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-21T14:59:53.611 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-21T14:59:53.625 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T14:59:54.057 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:54.056030 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-21T14:59:54.207 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:54.205447 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-21T14:59:54.501 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:59:54.502 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:49.000000Z", "last_refresh": "2022-03-21T14:59:54.162155Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T14:59:54.162155Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T14:59:54.162155Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T14:59:54.162155Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T14:59:54.162155Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T14:59:54.162155Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-21T14:59:54.162155Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T14:59:54.162155Z", "running": 0, "size": 1}}] 2022-03-21T14:59:54.525 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch rm mds.foo 2022-03-21T14:59:55.077 INFO:teuthology.orchestra.run.smithi043.stdout:Removed cephfilesystems/foo 2022-03-21T14:59:55.099 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch rm rbd-mirror 2022-03-21T14:59:55.629 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:55.628294 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-21T14:59:55.839 INFO:teuthology.orchestra.run.smithi043.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-21T14:59:55.858 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch rm rgw.foo 2022-03-21T14:59:55.963 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:55.962210 I | ceph-object-controller: reconciling object store deployments 2022-03-21T14:59:56.027 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:56.025753 I | ceph-object-controller: ceph object store gateway service running at 10.103.233.169 2022-03-21T14:59:56.027 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:56.025775 I | ceph-object-controller: reconciling object store pools 2022-03-21T14:59:56.121 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:56.038525 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-21T14:59:56.333 INFO:teuthology.orchestra.run.smithi043.stdout:Removed cephobjectstores/foo 2022-03-21T14:59:56.351 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T14:59:56.353 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:56.352152 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-21T14:59:57.303 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:57.301594 I | op-mds: setting mds config flags 2022-03-21T14:59:57.303 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:57.301624 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-21T14:59:57.389 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:57.387986 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-21T14:59:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T14:59:57.490 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:49.000000Z", "last_refresh": "2022-03-21T14:59:57.269575Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T14:59:57.269575Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T14:59:57.269575Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T14:59:57.269575Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T14:59:57.269575Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T14:59:57.269575Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T14:59:57.269575Z", "running": 0, "size": 1}}] 2022-03-21T14:59:57.783 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:57.781209 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-21T14:59:57.783 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:57.781234 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-21T14:59:57.784 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:57.781243 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-21T14:59:57.927 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:57.925536 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-21T14:59:57.937 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:57.936003 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-21T14:59:59.431 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:59.429645 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-21T14:59:59.673 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:59.671485 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-21T14:59:59.803 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 14:59:59.801448 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi065": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T15:00:00.602 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:00.600848 I | op-mds: setting mds config flags 2022-03-21T15:00:00.603 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:00.600872 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-21T15:00:03.395 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:03.393598 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-21T15:00:05.018 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:05.017196 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-21T15:00:05.023 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:05.021722 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-21T15:00:05.477 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:05.475549 I | ceph-file-controller: creating filesystem "foo" 2022-03-21T15:00:06.698 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:06.696998 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-21T15:00:06.789 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:06.787808 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-21T15:00:06.792 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:06.789946 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-21T15:00:07.508 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:00:08.457 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:00:08.458 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:49.000000Z", "last_refresh": "2022-03-21T15:00:08.256520Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:00:08.256520Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:00:08.256520Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:00:08.256520Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:00:08.256520Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:00:08.256520Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:00:08.256520Z", "running": 0, "size": 1}}] 2022-03-21T15:00:08.483 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:08.482657 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-21T15:00:08.498 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:08.496673 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:11.304 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:11.302920 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-21T15:00:13.091 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:13.089587 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:13.079+0000 7f343b710380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:13.092 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:13.079+0000 7f343b710380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:13.092 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:13.079+0000 7f343b710380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:13.092 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:13.131 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:13.129593 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:13.775 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:13.773039 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:13.763+0000 7f7ed983a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:13.775 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:13.763+0000 7f7ed983a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:13.775 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:13.763+0000 7f7ed983a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:13.776 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:13.802 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:13.800293 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:14.474 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:14.473021 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:14.467+0000 7f414ac51380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:14.475 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:14.467+0000 7f414ac51380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:14.475 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:14.467+0000 7f414ac51380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:14.475 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:14.509 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:14.508385 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:15.082 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:15.080186 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:15.071+0000 7f99de601380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:15.082 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:15.071+0000 7f99de601380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:15.083 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:15.071+0000 7f99de601380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:15.083 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:15.128 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:15.126367 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:15.248 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:15.246740 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-21T15:00:15.726 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:15.724648 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:15.719+0000 7f4193ac2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:15.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:15.719+0000 7f4193ac2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:15.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:15.719+0000 7f4193ac2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:15.727 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:15.739 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:15.738463 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:15.943 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:15.942126 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-21T15:00:16.357 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:16.352588 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:16.343+0000 7f29ebe6a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:16.358 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:16.343+0000 7f29ebe6a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:16.358 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:16.343+0000 7f29ebe6a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:16.358 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:16.364 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:16.363389 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:17.009 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:17.007759 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:16.999+0000 7f758dc76380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:17.010 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:16.999+0000 7f758dc76380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:17.010 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:16.999+0000 7f758dc76380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:17.010 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:17.016 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:17.015130 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:17.651 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:17.649417 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:17.643+0000 7fa667ce5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:17.652 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:17.643+0000 7fa667ce5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:17.652 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:17.643+0000 7fa667ce5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:17.652 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:17.664 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:17.663103 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:18.088 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:18.086652 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-21T15:00:18.088 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:18.086678 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-21T15:00:18.302 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:18.300890 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:18.295+0000 7fa54b501380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:18.303 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:18.295+0000 7fa54b501380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:18.303 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:18.295+0000 7fa54b501380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:18.303 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:18.478 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:00:18.939 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:18.938200 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:19.183 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:00:19.183 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:00:19.013326Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:00:19.013326Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:00:19.013326Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:00:19.013326Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:00:19.013326Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:00:19.013326Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:00:19.013326Z", "running": 0, "size": 1}}] 2022-03-21T15:00:19.373 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:19.372029 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:19.378 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:19.373899 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-21T15:00:19.560 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:19.557838 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:19.551+0000 7fc7b2ae4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:19.560 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:19.551+0000 7fc7b2ae4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:19.560 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:19.551+0000 7fc7b2ae4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:19.561 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:20.227 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:20.225836 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-21T15:00:20.713 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:20.711225 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-21T15:00:20.713 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:20.711255 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-21T15:00:20.713 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:20.711262 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-21T15:00:21.243 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:21.241676 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-21T15:00:21.606 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:21.605151 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-21T15:00:22.089 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:22.087343 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-21T15:00:22.089 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:22.087374 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-21T15:00:22.090 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:22.087381 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-21T15:00:22.816 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:22.814694 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-21T15:00:22.817 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:22.814717 I | ceph-file-controller: downing filesystem "foo" 2022-03-21T15:00:24.074 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:24.072264 I | ceph-file-controller: downed filesystem "foo" 2022-03-21T15:00:24.687 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:24.685551 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:25.796 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:25.793834 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:25.783+0000 7f7e03849380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:25.796 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:25.783+0000 7f7e03849380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:25.796 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:25.783+0000 7f7e03849380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:25.796 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:27.568 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:27.566676 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-21T15:00:27.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:27.566701 I | cephclient: purging pool "foo-metadata" (id=8) 2022-03-21T15:00:29.216 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:00:29.376 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:29.373715 I | cephclient: purge completed for pool "foo-metadata" 2022-03-21T15:00:29.833 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:29.831374 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-21T15:00:29.833 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:29.831397 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-21T15:00:30.123 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:00:30.125 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:00:29.863715Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:00:29.863715Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:00:29.863715Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:00:29.863715Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:00:29.863715Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:00:29.863715Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:00:29.863715Z", "running": 0, "size": 1}}] 2022-03-21T15:00:31.919 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:31.918005 I | cephclient: purge completed for pool "foo-data0" 2022-03-21T15:00:31.920 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:31.918102 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-21T15:00:31.987 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:31.986342 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-21T15:00:31.997 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:31.995912 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-21T15:00:32.000 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:31.996682 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-21T15:00:32.001 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:31.996785 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-21T15:00:35.095 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:35.093843 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi154": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T15:00:36.167 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:36.165790 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:37.219 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:37.216491 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:37.206+0000 7f3231126380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:37.219 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:37.206+0000 7f3231126380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:37.220 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:37.206+0000 7f3231126380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:37.220 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:00:40.147 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:00:40.877 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:00:40.878 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:00:40.697632Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:00:40.697632Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:00:40.697632Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:00:40.697632Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:00:40.697632Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:00:40.697632Z", "running": 0, "size": 1}}] 2022-03-21T15:00:50.910 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:00:54.298 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:00:54.299 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:00:54.079675Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:00:54.079675Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:00:54.079675Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:00:54.079675Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:00:54.079675Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:00:54.079675Z", "running": 0, "size": 1}}] 2022-03-21T15:00:57.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:57.705080 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:00:58.456 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:00:58.454314 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:00:58.442+0000 7fa56f501380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:00:58.456 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:58.442+0000 7fa56f501380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:00:58.456 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:00:58.442+0000 7fa56f501380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:00:58.457 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:01:04.327 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:01:05.039 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:01:05.039 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:01:04.877822Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:01:04.877822Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:01:04.877822Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:01:04.877822Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:01:04.877822Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:01:04.877822Z", "running": 0, "size": 1}}] 2022-03-21T15:01:15.068 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:01:15.749 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:01:15.750 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:01:15.592721Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:01:15.592721Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:01:15.592721Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:01:15.592721Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:01:15.592721Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:01:15.592721Z", "running": 0, "size": 1}}] 2022-03-21T15:01:25.775 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:01:26.492 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:01:26.492 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:01:26.309009Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:01:26.309009Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:01:26.309009Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:01:26.309009Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:01:26.309009Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:01:26.309009Z", "running": 0, "size": 1}}] 2022-03-21T15:01:36.553 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:01:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:01:37.297 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:01:37.108702Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:01:37.108702Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:01:37.108702Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:01:37.108702Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:01:37.108702Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:01:37.108702Z", "running": 0, "size": 1}}] 2022-03-21T15:01:39.431 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:01:39.429287 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:01:40.138 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:01:40.136493 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:01:40.125+0000 7f1d6ad74380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:01:40.139 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:01:40.125+0000 7f1d6ad74380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:01:40.139 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:01:40.125+0000 7f1d6ad74380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:01:40.139 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:01:47.322 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:01:48.092 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:01:48.093 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:01:47.902528Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:01:47.902528Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:01:47.902528Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:01:47.902528Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:01:47.902528Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:01:47.902528Z", "running": 0, "size": 1}}] 2022-03-21T15:01:58.113 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:01:58.847 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:01:58.848 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:01:58.669720Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:01:58.669720Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:01:58.669720Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:01:58.669720Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:01:58.669720Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:01:58.669720Z", "running": 0, "size": 1}}] 2022-03-21T15:02:04.360 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:02:04.359192 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi065": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T15:02:04.635 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:02:04.633979 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi065": the object has been modified; please apply your changes to the latest version and try again 2022-03-21T15:02:08.894 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:02:09.649 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:02:09.650 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:02:09.480980Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:02:09.480980Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:02:09.480980Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:02:09.480980Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:02:09.480980Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:02:09.480980Z", "running": 0, "size": 1}}] 2022-03-21T15:02:19.675 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:02:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:02:20.353 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:02:20.185758Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:02:20.185758Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:02:20.185758Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:02:20.185758Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:02:20.185758Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:02:20.185758Z", "running": 0, "size": 1}}] 2022-03-21T15:02:30.382 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:02:31.091 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:02:31.092 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:02:30.933988Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:02:30.933988Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:02:30.933988Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:02:30.933988Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:02:30.933988Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:02:30.933988Z", "running": 0, "size": 1}}] 2022-03-21T15:02:41.117 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:02:41.844 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:02:41.845 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:02:41.670605Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:02:41.670605Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:02:41.670605Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:02:41.670605Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:02:41.670605Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:02:41.670605Z", "running": 0, "size": 1}}] 2022-03-21T15:02:51.865 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:02:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:02:52.677 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:02:52.429920Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:02:52.429920Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:02:52.429920Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:02:52.429920Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:02:52.429920Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:02:52.429920Z", "running": 0, "size": 1}}] 2022-03-21T15:03:02.066 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:03:02.064606 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:03:02.703 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:03:02.856 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:03:02.854106 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:03:02.843+0000 7fcf81e80380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:03:02.856 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:03:02.843+0000 7fcf81e80380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:03:02.856 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:03:02.843+0000 7fcf81e80380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:03:02.857 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:03:03.692 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:03:03.692 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:03:03.255324Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:03:03.255324Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:03:03.255324Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:03:03.255324Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:03:03.255324Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:03:03.255324Z", "running": 0, "size": 1}}] 2022-03-21T15:03:13.714 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:03:14.458 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:03:14.459 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:03:14.301264Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:03:14.301264Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:03:14.301264Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:03:14.301264Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:03:14.301264Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:03:14.301264Z", "running": 0, "size": 1}}] 2022-03-21T15:03:24.489 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:03:25.176 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:03:25.177 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:03:25.016467Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:03:25.016467Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:03:25.016467Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:03:25.016467Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:03:25.016467Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:03:25.016467Z", "running": 0, "size": 1}}] 2022-03-21T15:03:35.209 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:03:35.983 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:03:35.983 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:03:35.727113Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:03:35.727113Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:03:35.727113Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:03:35.727113Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:03:35.727113Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:03:35.727113Z", "running": 0, "size": 1}}] 2022-03-21T15:03:46.013 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:03:46.711 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:03:46.712 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:03:46.552058Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:03:46.552058Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:03:46.552058Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:03:46.552058Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:03:46.552058Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:03:46.552058Z", "running": 0, "size": 1}}] 2022-03-21T15:03:56.734 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:03:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:03:57.474 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:03:57.286477Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:03:57.286477Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:03:57.286477Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:03:57.286477Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:03:57.286477Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:03:57.286477Z", "running": 0, "size": 1}}] 2022-03-21T15:04:07.502 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:04:08.468 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:04:08.469 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:04:08.029426Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:04:08.029426Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:04:08.029426Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:04:08.029426Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:04:08.029426Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:04:08.029426Z", "running": 0, "size": 1}}] 2022-03-21T15:04:18.504 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:04:19.183 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:04:19.183 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:04:19.021588Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:04:19.021588Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:04:19.021588Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:04:19.021588Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:04:19.021588Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:04:19.021588Z", "running": 0, "size": 1}}] 2022-03-21T15:04:29.217 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:04:29.948 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:04:29.949 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:04:29.780532Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:04:29.780532Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:04:29.780532Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:04:29.780532Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:04:29.780532Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:04:29.780532Z", "running": 0, "size": 1}}] 2022-03-21T15:04:39.976 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:04:40.681 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:04:40.682 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:04:40.507783Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:04:40.507783Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:04:40.507783Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:04:40.507783Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:04:40.507783Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:04:40.507783Z", "running": 0, "size": 1}}] 2022-03-21T15:04:50.708 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:04:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:04:51.469 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:04:51.280638Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:04:51.280638Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:04:51.280638Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:04:51.280638Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:04:51.280638Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:04:51.280638Z", "running": 0, "size": 1}}] 2022-03-21T15:05:01.495 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:05:02.229 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:05:02.230 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:05:02.058221Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:05:02.058221Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:05:02.058221Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:05:02.058221Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:05:02.058221Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:05:02.058221Z", "running": 0, "size": 1}}] 2022-03-21T15:05:12.255 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:05:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:05:12.951 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:05:12.782080Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:05:12.782080Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:05:12.782080Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:05:12.782080Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:05:12.782080Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:05:12.782080Z", "running": 0, "size": 1}}] 2022-03-21T15:05:22.976 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:05:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:05:23.707 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:05:23.525695Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:05:23.525695Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:05:23.525695Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:05:23.525695Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:05:23.525695Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:05:23.525695Z", "running": 0, "size": 1}}] 2022-03-21T15:05:33.732 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:05:34.445 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:05:34.446 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:05:34.280861Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:05:34.280861Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:05:34.280861Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:05:34.280861Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:05:34.280861Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:05:34.280861Z", "running": 0, "size": 1}}] 2022-03-21T15:05:44.473 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:05:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:05:45.190 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:05:44.986164Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:05:44.986164Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:05:44.986164Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:05:44.986164Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:05:44.986164Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:05:44.986164Z", "running": 0, "size": 1}}] 2022-03-21T15:05:46.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:05:46.704356 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:05:47.413 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:05:47.411660 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:05:47.400+0000 7fc70e249380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:05:47.414 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:05:47.400+0000 7fc70e249380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:05:47.414 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:05:47.400+0000 7fc70e249380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:05:47.414 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:05:55.222 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:05:55.943 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:05:55.944 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:05:55.763513Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:05:55.763513Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:05:55.763513Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:05:55.763513Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:05:55.763513Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:05:55.763513Z", "running": 0, "size": 1}}] 2022-03-21T15:06:05.962 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:06:06.656 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:06:06.656 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:06:06.468109Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:06:06.468109Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:06:06.468109Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:06:06.468109Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:06:06.468109Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:06:06.468109Z", "running": 0, "size": 1}}] 2022-03-21T15:06:16.686 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:06:17.402 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:06:17.402 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:06:17.226479Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:06:17.226479Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:06:17.226479Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:06:17.226479Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:06:17.226479Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:06:17.226479Z", "running": 0, "size": 1}}] 2022-03-21T15:06:27.430 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:06:28.130 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:06:28.131 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:06:27.950462Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:06:27.950462Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:06:27.950462Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:06:27.950462Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:06:27.950462Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:06:27.950462Z", "running": 0, "size": 1}}] 2022-03-21T15:06:38.163 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:06:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:06:38.951 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:06:38.738758Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:06:38.738758Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:06:38.738758Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:06:38.738758Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:06:38.738758Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:06:38.738758Z", "running": 0, "size": 1}}] 2022-03-21T15:06:48.978 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:06:49.699 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:06:49.700 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:06:49.502013Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:06:49.502013Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:06:49.502013Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:06:49.502013Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:06:49.502013Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:06:49.502013Z", "running": 0, "size": 1}}] 2022-03-21T15:06:59.728 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:07:00.537 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:07:00.538 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:07:00.367530Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:07:00.367530Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:07:00.367530Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:07:00.367530Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:07:00.367530Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:07:00.367530Z", "running": 0, "size": 1}}] 2022-03-21T15:07:10.566 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:07:11.275 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:07:11.276 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:07:11.114989Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:07:11.114989Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:07:11.114989Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:07:11.114989Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:07:11.114989Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:07:11.114989Z", "running": 0, "size": 1}}] 2022-03-21T15:07:21.303 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:07:22.028 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:07:22.029 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:07:21.849450Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:07:21.849450Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:07:21.849450Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:07:21.849450Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:07:21.849450Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:07:21.849450Z", "running": 0, "size": 1}}] 2022-03-21T15:07:32.055 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:07:32.808 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:07:32.808 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:07:32.644368Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:07:32.644368Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:07:32.644368Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:07:32.644368Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:07:32.644368Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:07:32.644368Z", "running": 0, "size": 1}}] 2022-03-21T15:07:42.833 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:07:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:07:43.537 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:07:43.354430Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:07:43.354430Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:07:43.354430Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:07:43.354430Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:07:43.354430Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:07:43.354430Z", "running": 0, "size": 1}}] 2022-03-21T15:07:53.571 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:07:54.323 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:07:54.324 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:07:54.110396Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:07:54.110396Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:07:54.110396Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:07:54.110396Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:07:54.110396Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:07:54.110396Z", "running": 0, "size": 1}}] 2022-03-21T15:08:04.352 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:08:05.281 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:08:05.281 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:08:05.106490Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:08:05.106490Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:08:05.106490Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:08:05.106490Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:08:05.106490Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:08:05.106490Z", "running": 0, "size": 1}}] 2022-03-21T15:08:15.308 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:08:16.054 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:08:16.054 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:08:15.893189Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:08:15.893189Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:08:15.893189Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:08:15.893189Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:08:15.893189Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:08:15.893189Z", "running": 0, "size": 1}}] 2022-03-21T15:08:26.080 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:08:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:08:26.856 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:08:26.662297Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:08:26.662297Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:08:26.662297Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:08:26.662297Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:08:26.662297Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:08:26.662297Z", "running": 0, "size": 1}}] 2022-03-21T15:08:36.882 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:08:37.660 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:08:37.660 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:08:37.483793Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:08:37.483793Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:08:37.483793Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:08:37.483793Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:08:37.483793Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:08:37.483793Z", "running": 0, "size": 1}}] 2022-03-21T15:08:47.688 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:08:48.425 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:08:48.425 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:08:48.252619Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:08:48.252619Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:08:48.252619Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:08:48.252619Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:08:48.252619Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:08:48.252619Z", "running": 0, "size": 1}}] 2022-03-21T15:08:58.450 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:08:59.260 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:08:59.261 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:08:58.993670Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:08:58.993670Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:08:58.993670Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:08:58.993670Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:08:58.993670Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:08:58.993670Z", "running": 0, "size": 1}}] 2022-03-21T15:09:09.288 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:09:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:09:09.983 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:09:09.817817Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:09:09.817817Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:09:09.817817Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:09:09.817817Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:09:09.817817Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:09:09.817817Z", "running": 0, "size": 1}}] 2022-03-21T15:09:20.008 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:09:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:09:20.881 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:09:20.716435Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:09:20.716435Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:09:20.716435Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:09:20.716435Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:09:20.716435Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:09:20.716435Z", "running": 0, "size": 1}}] 2022-03-21T15:09:30.907 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:09:31.637 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:09:31.638 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:09:31.463394Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:09:31.463394Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:09:31.463394Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:09:31.463394Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:09:31.463394Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:09:31.463394Z", "running": 0, "size": 1}}] 2022-03-21T15:09:41.664 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:09:42.366 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:09:42.366 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:09:42.193037Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:09:42.193037Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:09:42.193037Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:09:42.193037Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:09:42.193037Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:09:42.193037Z", "running": 0, "size": 1}}] 2022-03-21T15:09:52.391 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:09:53.165 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:09:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:09:52.957472Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:09:52.957472Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:09:52.957472Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:09:52.957472Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:09:52.957472Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:09:52.957472Z", "running": 0, "size": 1}}] 2022-03-21T15:10:03.193 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:10:03.939 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:10:03.939 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:10:03.764546Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:10:03.764546Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:10:03.764546Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:10:03.764546Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:10:03.764546Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:10:03.764546Z", "running": 0, "size": 1}}] 2022-03-21T15:10:13.969 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:10:14.705 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:10:14.705 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:10:14.535514Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:10:14.535514Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:10:14.535514Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:10:14.535514Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:10:14.535514Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:10:14.535514Z", "running": 0, "size": 1}}] 2022-03-21T15:10:24.732 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:10:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:10:25.432 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:10:25.252681Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:10:25.252681Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:10:25.252681Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:10:25.252681Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:10:25.252681Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:10:25.252681Z", "running": 0, "size": 1}}] 2022-03-21T15:10:35.458 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:10:36.145 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:10:36.146 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:10:35.982284Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:10:35.982284Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:10:35.982284Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:10:35.982284Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:10:35.982284Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:10:35.982284Z", "running": 0, "size": 1}}] 2022-03-21T15:10:46.173 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:10:46.953 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:10:46.954 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:10:46.785943Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:10:46.785943Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:10:46.785943Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:10:46.785943Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:10:46.785943Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:10:46.785943Z", "running": 0, "size": 1}}] 2022-03-21T15:10:56.980 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:11:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:11:00.481 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:11:00.316491Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:11:00.316491Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:11:00.316491Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:11:00.316491Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:11:00.316491Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:11:00.316491Z", "running": 0, "size": 1}}] 2022-03-21T15:11:10.509 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:11:11.239 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:11:11.239 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:11:11.067024Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:11:11.067024Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:11:11.067024Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:11:11.067024Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:11:11.067024Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:11:11.067024Z", "running": 0, "size": 1}}] 2022-03-21T15:11:15.101 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:11:15.099043 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:11:15.787 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:11:15.785697 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:11:15.773+0000 7f2b695a9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:11:15.788 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:11:15.773+0000 7f2b695a9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:11:15.788 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:11:15.773+0000 7f2b695a9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:11:15.788 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:11:21.265 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:11:22.022 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:11:22.022 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:11:21.826309Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:11:21.826309Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:11:21.826309Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:11:21.826309Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:11:21.826309Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:11:21.826309Z", "running": 0, "size": 1}}] 2022-03-21T15:11:32.046 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:11:32.776 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:11:32.776 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:11:32.603140Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:11:32.603140Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:11:32.603140Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:11:32.603140Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:11:32.603140Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:11:32.603140Z", "running": 0, "size": 1}}] 2022-03-21T15:11:42.801 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:11:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:11:43.653 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:11:43.429540Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:11:43.429540Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:11:43.429540Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:11:43.429540Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:11:43.429540Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:11:43.429540Z", "running": 0, "size": 1}}] 2022-03-21T15:11:53.655 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:11:54.531 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:11:54.531 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:11:54.219275Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:11:54.219275Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:11:54.219275Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:11:54.219275Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:11:54.219275Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:11:54.219275Z", "running": 0, "size": 1}}] 2022-03-21T15:12:04.558 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:12:05.415 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:12:05.415 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:12:05.181301Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:12:05.181301Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:12:05.181301Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:12:05.181301Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:12:05.181301Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:12:05.181301Z", "running": 0, "size": 1}}] 2022-03-21T15:12:15.443 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:12:16.226 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:12:16.226 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:12:16.060339Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:12:16.060339Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:12:16.060339Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:12:16.060339Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:12:16.060339Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:12:16.060339Z", "running": 0, "size": 1}}] 2022-03-21T15:12:26.254 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:12:26.988 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:12:26.988 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:12:26.806082Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:12:26.806082Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:12:26.806082Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:12:26.806082Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:12:26.806082Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:12:26.806082Z", "running": 0, "size": 1}}] 2022-03-21T15:12:37.014 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:12:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:12:37.761 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:12:37.593231Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:12:37.593231Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:12:37.593231Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:12:37.593231Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:12:37.593231Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:12:37.593231Z", "running": 0, "size": 1}}] 2022-03-21T15:12:47.790 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:12:48.539 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:12:48.539 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:12:48.348434Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:12:48.348434Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:12:48.348434Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:12:48.348434Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:12:48.348434Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:12:48.348434Z", "running": 0, "size": 1}}] 2022-03-21T15:12:58.567 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:12:59.302 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:12:59.302 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:12:59.115514Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:12:59.115514Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:12:59.115514Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:12:59.115514Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:12:59.115514Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:12:59.115514Z", "running": 0, "size": 1}}] 2022-03-21T15:13:09.331 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:13:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:13:10.017 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:13:09.847287Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:13:09.847287Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:13:09.847287Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:13:09.847287Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:13:09.847287Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:13:09.847287Z", "running": 0, "size": 1}}] 2022-03-21T15:13:20.040 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:13:20.799 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:13:20.799 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:13:20.622516Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:13:20.622516Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:13:20.622516Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:13:20.622516Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:13:20.622516Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:13:20.622516Z", "running": 0, "size": 1}}] 2022-03-21T15:13:30.826 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:13:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:13:31.616 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:13:31.444991Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:13:31.444991Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:13:31.444991Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:13:31.444991Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:13:31.444991Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:13:31.444991Z", "running": 0, "size": 1}}] 2022-03-21T15:13:41.644 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:13:42.333 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:13:42.334 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:13:42.169797Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:13:42.169797Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:13:42.169797Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:13:42.169797Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:13:42.169797Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:13:42.169797Z", "running": 0, "size": 1}}] 2022-03-21T15:13:52.361 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:13:53.131 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:13:53.132 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:13:52.928564Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:13:52.928564Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:13:52.928564Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:13:52.928564Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:13:52.928564Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:13:52.928564Z", "running": 0, "size": 1}}] 2022-03-21T15:14:03.156 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:14:03.926 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:14:03.927 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:14:03.750894Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:14:03.750894Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:14:03.750894Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:14:03.750894Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:14:03.750894Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:14:03.750894Z", "running": 0, "size": 1}}] 2022-03-21T15:14:13.950 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:14:14.665 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:14:14.666 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:14:14.491184Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:14:14.491184Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:14:14.491184Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:14:14.491184Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:14:14.491184Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:14:14.491184Z", "running": 0, "size": 1}}] 2022-03-21T15:14:24.691 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:14:25.397 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:14:25.397 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:14:25.214697Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:14:25.214697Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:14:25.214697Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:14:25.214697Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:14:25.214697Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:14:25.214697Z", "running": 0, "size": 1}}] 2022-03-21T15:14:35.424 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:14:36.204 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:14:36.205 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:14:36.011150Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:14:36.011150Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:14:36.011150Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:14:36.011150Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:14:36.011150Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:14:36.011150Z", "running": 0, "size": 1}}] 2022-03-21T15:14:46.231 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:14:46.961 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:14:46.962 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:14:46.794681Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:14:46.794681Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:14:46.794681Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:14:46.794681Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:14:46.794681Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:14:46.794681Z", "running": 0, "size": 1}}] 2022-03-21T15:14:56.985 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:14:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:14:57.716 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:14:57.510463Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:14:57.510463Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:14:57.510463Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:14:57.510463Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:14:57.510463Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:14:57.510463Z", "running": 0, "size": 1}}] 2022-03-21T15:15:07.744 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:15:08.490 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:15:08.491 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:15:08.329858Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:15:08.329858Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:15:08.329858Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:15:08.329858Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:15:08.329858Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:15:08.329858Z", "running": 0, "size": 1}}] 2022-03-21T15:15:18.517 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:15:19.235 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:15:19.236 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:15:19.057944Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:15:19.057944Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:15:19.057944Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:15:19.057944Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:15:19.057944Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:15:19.057944Z", "running": 0, "size": 1}}] 2022-03-21T15:15:29.263 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:15:30.089 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:15:30.090 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:15:29.821606Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:15:29.821606Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:15:29.821606Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:15:29.821606Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:15:29.821606Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:15:29.821606Z", "running": 0, "size": 1}}] 2022-03-21T15:15:40.117 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:15:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:15:40.830 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:15:40.661226Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:15:40.661226Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:15:40.661226Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:15:40.661226Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:15:40.661226Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:15:40.661226Z", "running": 0, "size": 1}}] 2022-03-21T15:15:50.856 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:15:51.595 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:15:51.596 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:15:51.435061Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:15:51.435061Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:15:51.435061Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:15:51.435061Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:15:51.435061Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:15:51.435061Z", "running": 0, "size": 1}}] 2022-03-21T15:16:01.623 DEBUG:teuthology.orchestra.run.smithi043:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-8rlv4 -- ceph orch ls -f json 2022-03-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-03-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi043.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:08.000000Z", "last_refresh": "2022-03-21T15:16:02.169434Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:54:44.000000Z", "last_refresh": "2022-03-21T15:16:02.169434Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:52:19.000000Z", "last_refresh": "2022-03-21T15:16:02.169434Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-21T14:56:04.000000Z", "last_refresh": "2022-03-21T15:16:02.169434Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-21T15:16:02.169434Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-21T15:16:02.169434Z", "running": 0, "size": 1}}] 2022-03-21T15:16:02.376 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-21T15:16:02.376 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-21T15:16:02.377 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-21T15:16:02.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-21T15:16:02.393 DEBUG:teuthology.orchestra.run.smithi065:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-21T15:16:02.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-21T15:16:02.426 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-21T15:16:02.427 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-21T15:16:02.437 DEBUG:teuthology.orchestra.remote:smithi043:rook/deploy/examples/operator.yaml is 20KB 2022-03-21T15:16:02.452 DEBUG:teuthology.orchestra.run.smithi043:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-21T15:16:02.516 INFO:teuthology.orchestra.run.smithi043.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-21T15:16:02.631 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-21T15:16:02.631 DEBUG:teuthology.orchestra.run.smithi043:> kubectl delete -f cluster.yaml 2022-03-21T15:16:02.706 INFO:teuthology.orchestra.run.smithi043.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-21T15:16:02.711 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:02.710152 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-21T15:16:02.712 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:02.710319 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-21T15:16:02.716 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:02.710736 I | operator: successfully started the controller-runtime manager 2022-03-21T15:16:02.716 INFO:tasks.rook.operator.smithi043.stdout:I0321 15:16:02.710760 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-21T15:16:02.795 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:02.793350 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-21T15:16:02.795 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:02.793382 I | operator: watching all namespaces for Ceph CRs 2022-03-21T15:16:02.796 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:02.793437 I | operator: setting up schemes 2022-03-21T15:16:02.800 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:02.797480 I | operator: setting up the controller-runtime manager 2022-03-21T15:16:03.003 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.001006 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-21T15:16:03.020 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017412 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-21T15:16:03.021 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017449 I | ceph-cluster-controller: successfully started 2022-03-21T15:16:03.021 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017494 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-21T15:16:03.021 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017508 I | ceph-crashcollector-controller: successfully started 2022-03-21T15:16:03.021 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017523 I | ceph-block-pool-controller: successfully started 2022-03-21T15:16:03.021 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017539 I | ceph-object-store-user-controller: successfully started 2022-03-21T15:16:03.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017552 I | ceph-object-realm-controller: successfully started 2022-03-21T15:16:03.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017561 I | ceph-object-zonegroup-controller: successfully started 2022-03-21T15:16:03.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017568 I | ceph-object-zone-controller: successfully started 2022-03-21T15:16:03.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017693 I | ceph-object-controller: successfully started 2022-03-21T15:16:03.023 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017719 I | ceph-file-controller: successfully started 2022-03-21T15:16:03.023 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017741 I | ceph-nfs-controller: successfully started 2022-03-21T15:16:03.023 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017759 I | ceph-rbd-mirror-controller: successfully started 2022-03-21T15:16:03.023 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017775 I | ceph-client-controller: successfully started 2022-03-21T15:16:03.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017790 I | ceph-filesystem-mirror-controller: successfully started 2022-03-21T15:16:03.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017808 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-21T15:16:03.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017830 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-21T15:16:03.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017847 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-21T15:16:03.025 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017867 I | ceph-bucket-topic: successfully started 2022-03-21T15:16:03.025 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017878 I | ceph-bucket-notification: successfully started 2022-03-21T15:16:03.025 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017885 I | ceph-bucket-notification: successfully started 2022-03-21T15:16:03.025 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.017894 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-21T15:16:03.026 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.019098 I | operator: starting the controller-runtime manager 2022-03-21T15:16:03.123 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.121145 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-21T15:16:03.123 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.121201 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-21T15:16:03.123 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.121216 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-21T15:16:03.128 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.125742 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-21T15:16:03.131 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.128954 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:03.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.198247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:03.211 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.209819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:03.789 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.787311 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:03.778+0000 7f052fd55380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:03.789 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:03.778+0000 7f052fd55380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:03.790 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:03.778+0000 7f052fd55380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:03.790 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:03.812 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:03.811269 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:04.479 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:04.477668 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:04.466+0000 7f32f3c80380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:04.480 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:04.466+0000 7f32f3c80380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:04.480 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:04.466+0000 7f32f3c80380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:04.480 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:04.506 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:04.505261 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:05.215 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:05.213514 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:05.202+0000 7fec32917380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:05.216 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:05.202+0000 7fec32917380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:05.216 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:05.202+0000 7fec32917380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:05.216 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:05.254 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:05.252783 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:06.001 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:05.999573 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:05.990+0000 7f0f52284380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:06.002 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:05.990+0000 7f0f52284380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:06.002 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:05.990+0000 7f0f52284380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:06.002 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:06.059 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:06.057554 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:06.767 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:06.764773 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:06.754+0000 7f1b4742f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:06.767 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:06.754+0000 7f1b4742f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:06.767 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:06.754+0000 7f1b4742f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:06.768 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:06.870 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:06.868913 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:07.598 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:07.596187 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:07.586+0000 7f2f49a4f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:07.598 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:07.586+0000 7f2f49a4f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:07.599 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:07.586+0000 7f2f49a4f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:07.599 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:07.765 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:07.763768 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:08.417 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:08.415933 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:08.406+0000 7fccd0855380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:08.418 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:08.406+0000 7fccd0855380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:08.418 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:08.406+0000 7fccd0855380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:08.418 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:08.746 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:08.744181 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:09.432 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:09.430063 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:09.418+0000 7f383f6cf380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:09.432 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:09.418+0000 7f383f6cf380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:09.432 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:09.418+0000 7f383f6cf380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:09.433 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:10.079 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:10.078069 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:10.733 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:10.731213 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:10.722+0000 7f833d2b4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:10.733 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:10.722+0000 7f833d2b4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:10.734 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:10.722+0000 7f833d2b4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:10.734 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:12.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:12.022453 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:12.681 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:12.679138 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:12.670+0000 7f8dbbac9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:12.681 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:12.670+0000 7f8dbbac9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:12.681 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:12.670+0000 7f8dbbac9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:12.682 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:13.279 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:13.277863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:13.293 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:13.291834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:15.248 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:15.246894 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:15.903 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:15.901991 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:15.890+0000 7f482e31b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:15.904 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:15.890+0000 7f482e31b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:15.904 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:15.890+0000 7f482e31b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:15.904 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:21.034 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:21.032593 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:21.792 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:21.790811 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:21.778+0000 7f5e0014c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:21.793 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:21.778+0000 7f5e0014c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:21.793 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:21.778+0000 7f5e0014c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:21.793 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:23.363 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:23.361369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:23.373 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:23.371711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:32.042 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:32.040652 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:32.914 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:32.912633 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:32.902+0000 7ff931339380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:32.915 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:32.902+0000 7ff931339380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:32.915 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:32.902+0000 7ff931339380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:32.915 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:16:33.472 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:33.470842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:33.486 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:33.484461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:43.596 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:43.595088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:43.611 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:43.608976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:53.540 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:53.538589 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:16:53.749 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:53.748042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:53.765 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:53.764023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:16:54.203 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:16:54.201326 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:16:54.189+0000 7fd733442380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:16:54.203 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:54.189+0000 7fd733442380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:16:54.204 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:16:54.189+0000 7fd733442380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:16:54.204 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:17:03.857 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:03.855490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:03.869 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:03.868108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:13.953 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:13.951669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:14.007 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:13.961402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:24.323 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:24.321037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:24.331 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:24.329778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:34.405 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:34.403978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:34.426 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:34.424637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:35.171 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:35.169962 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:17:35.845 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:35.843261 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:17:35.832+0000 7f8ac9346380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:17:35.845 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:17:35.832+0000 7f8ac9346380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:17:35.846 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:17:35.832+0000 7f8ac9346380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:17:35.846 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:17:44.535 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:44.533264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:44.550 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:44.548701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:54.654 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:54.652181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:17:54.665 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:17:54.663964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:04.746 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:04.744437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:04.798 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:04.797031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:14.893 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:14.891371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:14.908 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:14.906460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:25.066 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:25.065127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:25.080 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:25.079133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:35.162 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:35.160340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:35.173 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:35.170055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:45.289 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:45.287555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:45.303 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:45.301624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:55.376 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:55.374723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:55.391 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:55.389867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:18:57.776 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:57.774966 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:18:58.441 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:18:58.438934 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:18:58.427+0000 7f2ca6553380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:18:58.441 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:18:58.427+0000 7f2ca6553380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:18:58.441 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:18:58.427+0000 7f2ca6553380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:18:58.442 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:19:05.558 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:05.556025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:05.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:05.567803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:15.644 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:15.642795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:15.658 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:15.656570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:25.819 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:25.817118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:25.831 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:25.829890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:35.931 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:35.929671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:35.943 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:35.941741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:46.029 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:46.027598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:46.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:46.040141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:56.194 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:56.192249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:19:56.208 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:19:56.206557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:06.357 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:06.355890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:06.370 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:06.368531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:16.571 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:16.569420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:16.581 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:16.580311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:26.649 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:26.648003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:26.660 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:26.659336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:36.740 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:36.738494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:36.797 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:36.795884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:46.964 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:46.963263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:46.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:46.977032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:57.094 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:57.092240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:20:57.106 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:20:57.104763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:07.220 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:07.218845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:07.233 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:07.231633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:17.362 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:17.360331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:17.374 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:17.372794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:27.456 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:27.454913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:27.466 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:27.465078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:37.581 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:37.579543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:37.595 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:37.593492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:42.288 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:42.286264 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:21:42.967 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:42.965442 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:21:42.955+0000 7f7067fc7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:21:42.968 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:21:42.955+0000 7f7067fc7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:21:42.968 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:21:42.955+0000 7f7067fc7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:21:42.968 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:21:47.673 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:47.671468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:47.686 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:47.685226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:57.799 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:57.798117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:21:57.813 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:21:57.812377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:07.881 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:07.878972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:07.894 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:07.893003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:17.970 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:17.968269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:17.984 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:17.982284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:28.132 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:28.131049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:28.146 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:28.145215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:38.227 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:38.225909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:38.239 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:38.237769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:48.328 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:48.326588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:48.341 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:48.339746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:58.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:58.400434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:22:58.472 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:22:58.470563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:08.570 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:08.568700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:08.582 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:08.580981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:18.720 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:18.718861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:18.733 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:18.732328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:28.837 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:28.835705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:28.851 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:28.849415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:38.923 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:38.921515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:38.933 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:38.932269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:49.100 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:49.098902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:49.114 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:49.112563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:59.180 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:59.178375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:23:59.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:23:59.229767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:09.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:09.323139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:09.338 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:09.336992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:19.419 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:19.417805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:19.434 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:19.432516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:29.497 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:29.493116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:29.507 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:29.505971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:39.610 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:39.608438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:39.622 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:39.621521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:49.821 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:49.819842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:49.834 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:49.833206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:59.977 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:59.975236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:24:59.990 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:24:59.988890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:10.085 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:10.083336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:10.095 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:10.093424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:20.155 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:20.153331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:20.201 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:20.199993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:30.349 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:30.347742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:30.363 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:30.362375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:40.545 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:40.543811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:40.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:40.555510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:50.623 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:50.621763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:25:50.696 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:25:50.694469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:00.805 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:00.803556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:00.818 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:00.816948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:10.944 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:10.942697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:10.956 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:10.954819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:21.078 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:21.076576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:21.093 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:21.092163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:31.244 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:31.242356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:31.258 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:31.256249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:41.365 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:41.363684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:41.381 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:41.379600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:51.507 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:51.505577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:26:51.523 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:26:51.521745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:01.628 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:01.625960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:01.747 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:01.639581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:10.666 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:10.664790 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:27:11.331 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:11.329773 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:27:11.316+0000 7fc86cd17380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:27:11.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:27:11.316+0000 7fc86cd17380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:27:11.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:27:11.316+0000 7fc86cd17380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:27:11.332 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:27:11.738 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:11.737232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:11.807 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:11.805196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:21.953 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:21.951511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:21.967 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:21.966101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:32.069 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:32.067360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:32.079 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:32.078284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:42.176 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:42.174658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:42.189 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:42.187584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:52.311 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:52.309788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:27:52.327 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:27:52.326281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:02.433 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:02.431396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:02.446 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:02.445225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:12.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:12.568143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:12.581 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:12.580231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:22.690 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:22.688590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:22.704 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:22.702732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:32.808 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:32.806200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:32.839 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:32.838095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:42.953 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:42.951769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:42.966 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:42.965178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:53.066 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:53.065073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:28:53.078 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:28:53.076893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:03.233 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:03.231767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:03.247 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:03.246539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:13.381 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:13.380049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:13.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:13.393349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:23.460 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:23.459050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:23.475 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:23.473678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:33.576 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:33.574751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:33.589 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:33.588465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:43.694 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:43.692530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:43.707 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:43.706011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:53.775 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:53.774348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:29:53.794 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:29:53.793322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:03.900 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:03.898441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:03.914 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:03.913128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:13.986 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:13.984977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:14.045 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:14.043830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:24.188 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:24.187073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:24.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:24.199281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:34.349 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:34.347689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:34.362 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:34.361092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:44.476 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:44.474684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:44.490 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:44.489115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:54.639 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:54.638348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:30:54.652 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:30:54.651327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:04.731 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:04.729968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:04.744 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:04.743160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:14.857 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:14.855726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:14.872 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:14.871290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:24.942 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:24.940872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:24.993 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:24.991904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:35.058 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:35.057304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:35.070 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:35.069407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:45.180 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:45.178493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:45.193 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:45.192141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:55.310 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:55.308651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:31:55.320 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:31:55.319026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:05.427 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:05.425817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:05.447 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:05.442936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:15.603 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:15.600891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:15.615 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:15.614063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:25.868 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:25.866960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:25.881 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:25.880477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:35.956 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:35.955221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:35.969 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:35.968056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:46.057 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:46.055888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:46.070 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:46.068884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:56.145 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:56.143681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:32:56.159 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:32:56.157733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:06.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:06.286050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:06.302 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:06.301130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:16.372 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:16.370960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:16.386 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:16.385663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:26.524 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:26.522765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:26.538 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:26.536920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:36.646 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:36.644567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:36.662 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:36.660929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:46.726 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:46.725202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:46.738 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:46.737058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:56.803 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:56.801921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:33:56.816 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:33:56.814794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:06.879 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:06.878534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:06.928 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:06.927091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:17.051 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:17.050343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:17.065 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:17.064235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:27.174 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:27.173156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:27.188 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:27.187617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:37.266 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:37.264473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:37.281 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:37.279897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:47.388 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:47.386430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:47.397 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:47.395643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:57.513 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:57.511294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:34:57.526 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:34:57.525319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:07.593 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:07.591672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:07.607 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:07.605902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:17.707 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:17.705446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:17.720 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:17.719574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:27.796 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:27.794483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:27.879 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:27.877645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:37.946 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:37.944740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:37.958 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:37.957650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:48.029 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:48.027460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:48.042 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:48.041465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:58.153 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:58.152000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:35:58.167 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:35:58.166383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:08.271 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:08.270271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:08.283 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:08.282205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:18.365 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:18.363744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:18.377 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:18.376808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:28.477 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:28.475770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:28.491 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:28.490626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:38.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:38.557849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:38.571 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:38.570168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:48.703 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:48.702134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:48.719 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:48.718123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:58.787 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:58.786056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:36:58.799 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:36:58.797829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:08.862 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:08.860593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:08.894 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:08.893149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:18.991 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:18.989846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:19.003 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:19.002485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:29.092 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:29.090814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:29.111 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:29.110515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:39.215 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:39.213781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:39.227 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:39.226408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:49.319 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:49.318380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:49.328 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:49.327064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:59.455 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:59.454247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:37:59.464 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:37:59.463563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:06.700 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:06.698928 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:38:07.393 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:07.391911 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:38:07.382+0000 7fc85aba8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:38:07.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:38:07.382+0000 7fc85aba8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:38:07.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:38:07.382+0000 7fc85aba8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:38:07.394 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:38:09.529 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:09.528479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:09.543 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:09.542885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:19.661 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:19.659980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:19.675 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:19.674269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:29.777 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:29.776181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:29.790 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:29.789077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:39.871 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:39.869626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:39.884 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:39.883268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:50.014 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:50.012619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:38:50.025 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:38:50.023931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:00.181 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:00.179469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:00.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:00.193763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:10.295 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:10.294383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:10.310 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:10.309581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:20.377 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:20.376442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:20.451 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:20.450476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:30.528 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:30.526601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:30.543 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:30.542246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:40.725 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:40.723598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:40.734 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:40.733012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:50.824 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:50.822566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:39:50.845 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:39:50.837134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:00.943 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:00.941406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:00.954 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:00.952941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:11.051 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:11.050431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:11.067 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:11.066118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:21.188 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:21.186574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:21.201 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:21.199861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:31.273 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:31.271807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:31.322 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:31.321927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:41.413 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:41.412242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:41.428 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:41.427342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:51.496 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:51.495155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:40:51.512 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:40:51.511005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:01.576 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:01.574574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:01.589 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:01.588318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:11.685 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:11.683808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:11.696 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:11.694809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:21.825 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:21.824214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:21.835 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:21.834981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:31.904 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:31.902554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:31.958 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:31.956287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:42.056 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:42.055466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:42.068 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:42.067922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:52.151 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:52.149732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:41:52.164 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:41:52.163503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:02.269 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:02.268007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:02.281 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:02.280952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:12.411 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:12.410362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:12.423 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:12.422745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:22.485 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:22.483988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:22.548 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:22.547110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:32.697 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:32.695785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:32.710 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:32.709150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:42.820 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:42.818955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:42.835 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:42.834213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:52.928 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:52.926743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:42:52.938 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:42:52.937396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:03.008 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:03.007028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:03.018 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:03.016961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:13.094 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:13.093526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:13.108 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:13.107906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:23.232 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:23.231345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:23.268 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:23.266767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:33.384 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:33.382799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:33.396 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:33.395360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:43.465 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:43.464049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:43.519 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:43.518444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:53.643 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:53.641859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:43:53.657 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:43:53.656300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:03.730 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:03.728899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:03.747 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:03.746248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:13.813 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:13.811876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:13.826 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:13.825277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:23.939 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:23.938113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:23.953 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:23.952375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:34.028 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:34.027273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:34.094 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:34.093390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:44.161 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:44.159855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:44.171 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:44.170840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:54.277 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:54.275770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:44:54.288 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:44:54.287173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:04.447 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:04.445687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:04.463 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:04.462888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:14.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:14.567907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:14.584 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:14.583214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:24.705 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:24.704120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:24.721 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:24.720218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:34.797 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:34.796632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:34.809 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:34.808759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:44.917 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:44.916442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:44.930 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:44.929837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:55.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:55.021583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:45:55.034 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:45:55.033070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:05.124 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:05.122806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:05.135 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:05.134288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:15.239 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:15.238041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:15.250 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:15.249682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:25.379 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:25.378586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:25.391 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:25.390706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:35.465 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:35.464029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:35.480 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:35.479465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:45.548 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:45.546745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:45.562 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:45.561360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:55.655 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:55.653892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:46:55.669 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:46:55.668301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:05.762 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:05.761031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:05.776 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:05.775265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:15.880 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:15.878367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:15.894 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:15.893903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:26.000 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:25.999365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:26.016 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:26.014975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:36.114 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:36.113139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:36.128 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:36.126992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:46.238 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:46.236978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:46.249 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:46.248479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:56.323 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:56.321307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:47:56.335 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:47:56.334300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:06.409 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:06.407764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:06.423 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:06.422646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:16.529 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:16.527905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:16.538 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:16.537767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:26.644 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:26.643444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:26.655 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:26.654263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:36.721 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:36.720264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:36.734 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:36.732945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:46.802 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:46.800741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:46.816 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:46.815158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:56.949 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:56.947828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:48:56.961 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:48:56.960329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:07.067 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:07.066255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:07.081 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:07.080302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:17.150 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:17.149435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:17.206 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:17.205688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:27.272 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:27.271225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:27.288 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:27.286966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:37.368 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:37.367153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:37.382 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:37.381084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:47.512 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:47.510796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:47.526 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:47.525169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:57.649 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:57.648026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:49:57.662 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:49:57.661759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:07.730 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:07.729686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:07.777 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:07.776690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:17.899 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:17.898241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:17.912 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:17.911444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:27.975 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:27.973284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:28.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:28.040438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:38.154 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:38.153012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:38.167 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:38.166594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:48.236 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:48.234737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:48.249 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:48.248010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:58.352 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:58.351570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:50:58.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:50:58.393574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:08.463 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:08.462665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:08.478 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:08.477133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:18.588 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:18.586999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:18.602 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:18.600953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:28.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:28.707131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:28.724 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:28.723447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:38.856 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:38.855291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:38.867 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:38.866734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:49.118 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:49.116719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:49.131 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:49.130054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:59.263 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:59.262396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:51:59.273 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:51:59.272829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:09.336 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:09.335709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:09.349 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:09.348709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:19.434 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:19.433041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:19.445 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:19.444616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:29.532 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:29.531567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:29.548 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:29.547516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:39.624 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:39.623152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:39.639 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:39.638654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:49.720 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:49.718985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:49.735 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:49.733895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:59.809 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:59.807801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:52:59.821 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:52:59.820071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:09.898 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:09.897149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:09.911 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:09.910567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:19.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:19.977759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:19.991 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:19.990707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:30.064 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:30.062879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:30.075 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:30.074084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:40.134 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:40.132835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:40.187 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:40.185982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:50.307 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:50.306600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:53:50.320 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:53:50.319495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:00.420 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:00.419012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:00.433 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:00.432585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:10.500 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:10.498702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:10.557 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:10.556113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:20.686 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:20.684594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:20.700 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:20.699314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:30.800 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:30.798550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:30.812 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:30.810911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:40.927 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:40.926316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:40.943 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:40.942196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:47.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:47.400548 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T15:54:48.082 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:48.081034 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T15:54:48.068+0000 7f8aeb988380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T15:54:48.083 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:54:48.068+0000 7f8aeb988380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T15:54:48.083 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T15:54:48.068+0000 7f8aeb988380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T15:54:48.083 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T15:54:51.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:51.038321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:54:51.053 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:54:51.052540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:01.127 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:01.126523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:01.140 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:01.139042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:11.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:11.196011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:11.211 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:11.209884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:21.334 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:21.332888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:21.346 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:21.345411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:31.457 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:31.455737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:31.471 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:31.470177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:41.605 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:41.603903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:41.622 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:41.620949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:51.685 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:51.683961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:55:51.699 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:55:51.697667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:01.771 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:01.769619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:01.783 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:01.782480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:11.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:11.900350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:11.914 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:11.913621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:22.017 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:22.015675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:22.029 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:22.028626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:32.097 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:32.096211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:32.107 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:32.106512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:42.246 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:42.245014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:42.258 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:42.257976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:52.356 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:52.355449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:56:52.369 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:56:52.368541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:02.443 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:02.441716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:02.490 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:02.489180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:12.591 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:12.589992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:12.602 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:12.601641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:22.689 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:22.688330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:22.701 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:22.700618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:32.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:32.771436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:32.786 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:32.784893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:42.864 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:42.863192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:42.880 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:42.878912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:52.971 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:52.970017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:57:52.986 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:57:52.985464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:03.168 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:03.166917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:03.196 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:03.195297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:13.282 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:13.280833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:13.295 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:13.293861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:23.399 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:23.397940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:23.413 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:23.411852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:33.476 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:33.475530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:33.489 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:33.488301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:43.579 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:43.578270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:43.592 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:43.591010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:53.689 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:53.688317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:58:53.703 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:58:53.702118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:03.831 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:03.830512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:03.880 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:03.879568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:14.014 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:14.011337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:14.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:14.023151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:24.127 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:24.126202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:24.140 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:24.139427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:34.246 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:34.245381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:34.260 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:34.259007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:44.373 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:44.372174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:44.387 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:44.386278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:54.489 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:54.487656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T15:59:54.501 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 15:59:54.499777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:04.600 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:04.599007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:04.612 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:04.610952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:14.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:14.726232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:14.744 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:14.743346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:24.822 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:24.820530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:24.889 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:24.887890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:35.016 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:35.015012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:35.028 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:35.026864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:45.154 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:45.153437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:45.168 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:45.164700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:55.244 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:55.243070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:00:55.304 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:00:55.303588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:05.421 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:05.419772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:05.435 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:05.434133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:15.535 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:15.533858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:15.590 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:15.547078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:25.649 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:25.642256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:25.656 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:25.655761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:35.776 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:35.775345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:35.789 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:35.788209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:45.893 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:45.892356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:45.904 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:45.903286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:56.005 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:56.004605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:01:56.020 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:01:56.019396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:06.107 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:06.105752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:06.196 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:06.195753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:16.307 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:16.305988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:16.320 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:16.319254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:26.476 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:26.474797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:26.490 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:26.489167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:36.589 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:36.588026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:36.600 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:36.599718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:46.692 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:46.691638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:46.703 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:46.702512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:56.779 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:56.778304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:02:56.824 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:02:56.823080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:06.902 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:06.901403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:06.915 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:06.914513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:17.051 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:17.050498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:17.065 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:17.064241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:27.148 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:27.147152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:27.163 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:27.161913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:37.280 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:37.278556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:37.292 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:37.291154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:47.393 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:47.392417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:47.407 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:47.406087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:57.498 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:57.496525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:03:57.510 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:03:57.509394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:07.579 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:07.578368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:07.591 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:07.590452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:17.677 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:17.676295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:17.691 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:17.690629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:27.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:27.753144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:27.849 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:27.847702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:37.930 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:37.929534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:37.944 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:37.942713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:48.050 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:48.049163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:48.063 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:48.062302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:58.181 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:58.180278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:04:58.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:04:58.194518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:08.270 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:08.269566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:08.327 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:08.326477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:18.400 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:18.399423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:18.477 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:18.475941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:28.544 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:28.543676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:28.557 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:28.556836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:38.634 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:38.632855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:38.646 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:38.645372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:48.751 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:48.749738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:48.764 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:48.763609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:58.856 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:58.854730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:05:58.871 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:05:58.870933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:08.970 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:08.969511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:08.982 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:08.981437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:19.056 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:19.055623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:19.069 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:19.068039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:29.166 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:29.165274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:29.177 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:29.176673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:39.283 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:39.281182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:39.296 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:39.295296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:49.414 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:49.412687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:49.428 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:49.426935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:59.497 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:59.495734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:06:59.509 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:06:59.508774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:09.605 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:09.604451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:09.619 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:09.617954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:19.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:19.686894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:19.700 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:19.698990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:29.787 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:29.785905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:29.801 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:29.800409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:39.868 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:39.867640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:39.881 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:39.880517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:49.980 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:49.978781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:07:49.993 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:07:49.992946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:00.122 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:00.121072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:00.133 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:00.132213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:10.248 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:10.247007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:10.261 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:10.260464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:20.387 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:20.385679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:20.401 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:20.400395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:30.526 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:30.525022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:30.539 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:30.538434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:40.628 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:40.627431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:40.642 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:40.641416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:50.714 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:50.712638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:08:50.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:08:50.767768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:00.873 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:00.872163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:00.887 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:00.886209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:11.003 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:11.002214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:11.017 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:11.016455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:21.117 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:21.116040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:21.130 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:21.129817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:31.240 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:31.239190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:31.255 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:31.253679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:41.360 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:41.359134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:41.373 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:41.372880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:51.446 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:51.445532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:09:51.458 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:09:51.457507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:01.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:01.557681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:01.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:01.568747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:11.653 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:11.652339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:11.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:11.726416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:21.832 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:21.831004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:21.846 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:21.845619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:31.909 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:31.908381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:31.967 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:31.965176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:42.086 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:42.085411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:42.098 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:42.097146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:52.211 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:52.209965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:10:52.223 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:10:52.222800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:02.314 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:02.313200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:02.365 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:02.363915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:12.483 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:12.482277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:12.498 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:12.497282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:22.624 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:22.623391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:22.637 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:22.636690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:28.092 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:28.090822 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T16:11:28.762 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:28.760505 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T16:11:28.751+0000 7f57a4692380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T16:11:28.762 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T16:11:28.751+0000 7f57a4692380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T16:11:28.762 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T16:11:28.751+0000 7f57a4692380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T16:11:28.763 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T16:11:32.791 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:32.788851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:32.803 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:32.802052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:42.904 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:42.903210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:42.917 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:42.916029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:53.088 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:53.087081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:11:53.101 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:11:53.100316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:03.182 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:03.181138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:03.194 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:03.193619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:13.293 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:13.292307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:13.305 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:13.304466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:23.422 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:23.421059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:23.435 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:23.434092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:33.498 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:33.496893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:33.513 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:33.511158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:43.590 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:43.589616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:43.713 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:43.712221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:53.949 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:53.947877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:12:53.963 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:12:53.961967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:04.058 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:04.057458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:04.072 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:04.071118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:14.155 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:14.153950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:14.166 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:14.165678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:24.259 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:24.258029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:24.270 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:24.269625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:34.365 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:34.364442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:34.375 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:34.374055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:44.473 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:44.472453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:44.488 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:44.487321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:54.608 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:54.607042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:13:54.622 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:13:54.621195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:04.732 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:04.731268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:04.744 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:04.743467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:14.846 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:14.845040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:14.858 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:14.857652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:24.983 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:24.981794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:25.013 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:25.011990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:35.079 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:35.077273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:35.093 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:35.092818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:45.218 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:45.217512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:45.252 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:45.251503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:55.399 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:55.398056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:14:55.409 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:14:55.408818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:05.514 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:05.513305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:05.527 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:05.526346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:15.627 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:15.625980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:15.638 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:15.637615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:25.701 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:25.698250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:25.752 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:25.751536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:35.864 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:35.863236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:35.877 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:35.875753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:45.956 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:45.954965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:45.970 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:45.969443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:56.078 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:56.077541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:15:56.093 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:15:56.092020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:06.165 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:06.163813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:06.216 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:06.215329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:16.293 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:16.292125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:16.308 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:16.307557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:26.425 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:26.424394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:26.439 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:26.437881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:36.549 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:36.542915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:36.554 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:36.553051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:46.663 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:46.662560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:46.674 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:46.673929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:56.741 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:56.739811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:16:56.752 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:16:56.750988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:06.831 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:06.830427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:06.843 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:06.842662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:16.950 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:16.948870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:16.962 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:16.961457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:27.063 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:27.062426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:27.073 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:27.072798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:37.163 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:37.161357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:37.173 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:37.172658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:47.274 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:47.272810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:47.337 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:47.336159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:57.423 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:57.422227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:17:57.436 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:17:57.435423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:07.501 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:07.499734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:07.515 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:07.513686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:17.586 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:17.585577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:17.599 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:17.598367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:27.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:27.704828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:27.723 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:27.722034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:37.790 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:37.788451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:37.805 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:37.804644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:47.918 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:47.916655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:47.931 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:47.930258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:58.040 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:58.038985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:18:58.055 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:18:58.053997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:08.129 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:08.127842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:08.141 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:08.140748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:18.220 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:18.219276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:18.236 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:18.235424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:28.326 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:28.325463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:28.342 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:28.341029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:38.558 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:38.556928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:38.574 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:38.573164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:48.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:48.706860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:48.721 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:48.720009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:58.854 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:58.852761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:19:58.866 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:19:58.864949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:08.979 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:08.977392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:08.990 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:08.988812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:19.139 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:19.138164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:19.152 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:19.151275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:29.241 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:29.240538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:29.255 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:29.254748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:39.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:39.331607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:39.344 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:39.343419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:49.457 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:49.456134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:49.470 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:49.469646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:59.598 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:59.597055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:20:59.608 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:20:59.607019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:09.710 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:09.709136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:09.723 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:09.721808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:19.836 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:19.834953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:19.851 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:19.850242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:29.928 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:29.927310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:29.940 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:29.939604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:40.029 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:40.018739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:40.427 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:40.032974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:50.130 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:50.129000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:21:50.142 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:21:50.140838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:00.263 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:00.261670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:00.277 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:00.275925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:10.412 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:10.411006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:10.433 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:10.431668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:20.572 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:20.570662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:20.587 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:20.585842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:30.711 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:30.709878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:30.726 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:30.725431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:40.853 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:40.851664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:40.865 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:40.863938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:50.975 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:50.973143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:22:50.986 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:22:50.984862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:01.047 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:01.046515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:01.060 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:01.058694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:11.121 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:11.120418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:11.135 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:11.134770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:21.197 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:21.196078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:21.212 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:21.210549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:31.296 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:31.294759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:31.311 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:31.309862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:41.379 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:41.378071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:41.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:41.393172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:51.512 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:51.510781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:23:51.525 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:23:51.523711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:01.593 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:01.591355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:02.152 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:01.604446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:11.680 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:11.679436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:11.699 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:11.696313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:21.824 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:21.822441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:21.836 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:21.835123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:31.945 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:31.944280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:31.960 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:31.959596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:42.048 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:42.046645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:42.065 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:42.063676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:52.186 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:52.185187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:24:52.201 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:24:52.199962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:02.299 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:02.298062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:02.310 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:02.308966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:12.373 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:12.371701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:12.386 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:12.384877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:22.504 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:22.502927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:22.516 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:22.515781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:32.589 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:32.588122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:32.602 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:32.600650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:42.710 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:42.708938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:42.723 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:42.722512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:52.839 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:52.837735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:25:52.851 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:25:52.849768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:03.009 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:03.007934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:03.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:03.023597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:13.130 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:13.128642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:13.143 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:13.142213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:23.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:23.285931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:23.303 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:23.300059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:33.452 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:33.450643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:33.470 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:33.462822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:43.589 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:43.587967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:43.603 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:43.600458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:53.701 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:53.700108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:26:53.718 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:26:53.716421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:03.805 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:03.803520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:03.818 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:03.817570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:13.915 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:13.914187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:13.929 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:13.928210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:24.030 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:24.029019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:24.048 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:24.046953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:34.152 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:34.151109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:34.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:34.182573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:44.291 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:44.290548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:44.303 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:44.302661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:54.398 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:54.396636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:27:54.412 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:27:54.411620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:04.515 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:04.514288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:04.527 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:04.526557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:08.829 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:08.827804 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T16:28:09.524 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:09.522882 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T16:28:09.513+0000 7fdf2b102380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T16:28:09.525 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T16:28:09.513+0000 7fdf2b102380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T16:28:09.525 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T16:28:09.513+0000 7fdf2b102380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T16:28:09.525 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T16:28:14.621 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:14.620229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:14.633 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:14.632782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:24.746 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:24.745035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:24.759 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:24.758395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:34.902 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:34.900622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:34.915 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:34.914344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:45.052 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:45.050684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:45.064 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:45.062764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:55.158 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:55.156168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:28:55.169 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:28:55.168060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:05.297 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:05.293030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:05.309 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:05.308587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:15.388 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:15.386650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:15.467 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:15.465389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:25.539 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:25.538085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:25.551 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:25.550034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:35.658 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:35.656861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:35.673 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:35.672027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:45.744 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:45.742533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:45.756 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:45.754853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:55.829 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:55.827694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:29:55.886 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:29:55.884717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:05.975 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:05.974122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:06.051 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:06.049660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:16.128 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:16.126620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:16.140 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:16.139210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:26.265 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:26.263666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:26.280 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:26.279421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:36.406 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:36.405348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:36.436 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:36.435210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:46.554 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:46.552712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:46.566 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:46.564245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:56.627 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:56.626215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:30:56.639 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:30:56.638289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:06.757 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:06.756023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:06.770 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:06.768749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:16.851 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:16.849922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:16.949 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:16.948132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:27.046 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:27.045015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:27.057 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:27.056113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:37.134 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:37.133220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:37.184 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:37.183013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:47.253 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:47.251828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:47.266 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:47.265415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:57.380 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:57.378437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:31:57.422 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:31:57.421064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:07.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:07.567516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:07.583 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:07.582021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:17.661 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:17.659465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:17.673 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:17.672288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:27.767 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:27.766217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:27.783 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:27.781722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:37.843 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:37.842267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:37.857 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:37.856122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:47.930 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:47.928565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:47.942 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:47.940362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:58.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:58.039703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:32:58.054 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:32:58.052730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:08.125 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:08.123681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:08.137 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:08.135925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:18.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:18.207987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:18.223 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:18.221934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:28.328 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:28.326704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:28.338 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:28.337594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:38.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:38.400566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:38.415 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:38.414380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:48.524 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:48.522611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:48.537 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:48.536028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:58.676 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:58.675459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:33:58.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:33:58.686751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:08.793 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:08.791656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:08.805 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:08.803976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:18.902 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:18.900496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:18.914 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:18.912934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:29.055 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:29.053636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:29.064 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:29.063731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:39.180 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:39.178729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:39.192 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:39.191019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:49.318 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:49.317110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:49.330 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:49.328982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:59.439 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:59.437516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:34:59.451 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:34:59.449821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:09.522 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:09.520389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:09.534 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:09.532795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:19.653 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:19.651899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:19.667 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:19.665828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:29.799 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:29.797469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:29.811 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:29.809578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:39.879 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:39.877072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:39.908 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:39.891441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:49.955 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:49.953472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:35:49.969 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:35:49.967786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:00.072 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:00.070882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:00.086 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:00.084409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:10.190 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:10.187520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:10.201 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:10.200235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:20.320 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:20.319004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:20.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:20.330839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:30.469 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:30.467742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:30.481 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:30.479836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:40.580 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:40.578380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:40.646 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:40.590589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:50.663 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:50.661383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:36:50.675 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:36:50.674373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:00.744 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:00.739253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:00.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:00.771039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:10.873 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:10.871508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:10.888 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:10.886717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:20.967 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:20.966042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:20.979 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:20.978455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:31.047 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:31.045841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:31.062 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:31.061125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:41.171 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:41.170185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:41.187 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:41.186074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:51.292 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:51.290468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:37:51.307 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:37:51.306033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:01.389 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:01.387549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:01.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:01.401128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:11.511 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:11.509797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:11.525 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:11.524353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:21.621 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:21.619912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:21.639 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:21.637012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:31.745 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:31.743421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:31.758 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:31.757024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:41.887 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:41.885253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:41.898 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:41.896928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:52.049 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:52.047880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:38:52.064 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:38:52.062967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:02.133 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:02.132178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:02.147 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:02.145458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:12.272 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:12.270748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:12.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:12.285928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:22.356 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:22.354836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:22.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:22.400665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:32.476 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:32.474270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:32.489 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:32.487612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:42.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:42.557613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:42.572 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:42.571232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:52.653 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:52.652275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:39:52.667 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:39:52.666322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:02.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:02.767301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:02.780 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:02.779510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:12.848 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:12.847069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:12.863 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:12.861561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:22.940 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:22.937625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:22.955 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:22.953819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:33.170 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:33.168360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:33.184 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:33.183134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:43.294 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:43.292408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:43.307 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:43.305950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:53.388 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:53.387085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:40:53.403 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:40:53.402263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:03.533 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:03.531407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:03.547 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:03.545425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:13.618 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:13.616387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:13.631 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:13.630313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:23.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:23.752508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:23.768 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:23.766847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:33.830 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:33.828500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:33.843 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:33.842157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:43.969 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:43.967421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:44.001 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:43.999598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:54.100 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:54.099002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:41:54.114 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:41:54.112865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:04.270 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:04.268881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:04.282 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:04.280614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:14.393 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:14.391845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:14.407 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:14.406053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:24.506 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:24.504839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:24.525 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:24.523438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:34.628 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:34.626438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:34.641 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:34.639876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:44.742 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:44.741013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:44.764 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:44.763135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:54.897 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:54.895488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:42:54.910 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:42:54.908710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:05.021 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:05.019556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:05.035 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:05.034316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:15.173 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:15.171034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:15.186 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:15.184908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:25.297 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:25.295001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:25.312 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:25.310768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:35.411 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:35.408849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:35.425 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:35.423376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:45.529 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:45.528239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:45.542 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:45.540868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:55.605 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:55.603609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:43:55.618 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:43:55.616854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:05.696 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:05.694650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:05.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:05.704783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:15.766 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:15.764274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:15.779 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:15.777905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:25.892 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:25.890718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:25.906 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:25.904981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:36.005 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:36.003290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:36.018 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:36.016830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:46.121 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:46.119793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:46.136 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:46.134298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:49.531 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:49.529940 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T16:44:50.197 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:50.195423 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T16:44:50.187+0000 7f5d051e9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T16:44:50.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T16:44:50.187+0000 7f5d051e9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T16:44:50.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T16:44:50.187+0000 7f5d051e9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T16:44:50.199 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T16:44:56.243 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:56.241206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:44:56.255 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:44:56.253868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:06.335 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:06.333804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:06.348 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:06.346332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:16.408 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:16.406765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:16.420 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:16.418784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:26.481 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:26.479547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:26.532 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:26.530893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:36.653 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:36.651714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:36.663 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:36.661288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:46.736 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:46.734613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:46.745 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:46.743756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:56.851 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:56.849799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:45:56.865 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:45:56.863249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:06.971 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:06.969327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:06.984 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:06.982291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:17.059 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:17.056955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:17.113 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:17.111042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:27.174 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:27.172493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:27.187 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:27.186026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:37.290 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:37.287472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:37.313 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:37.303097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:47.409 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:47.407449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:47.422 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:47.419980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:57.503 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:57.501644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:46:57.515 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:46:57.513673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:07.590 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:07.588771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:07.602 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:07.601050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:17.701 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:17.692772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:17.835 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:17.833255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:27.902 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:27.900756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:27.914 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:27.912629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:37.994 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:37.992495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:38.006 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:38.004601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:48.071 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:48.069884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:48.131 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:48.129215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:58.241 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:58.239231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:47:58.252 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:47:58.251083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:08.322 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:08.320705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:08.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:08.331726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:18.404 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:18.402252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:18.417 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:18.415215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:28.531 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:28.528956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:28.542 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:28.540876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:38.669 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:38.664449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:38.681 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:38.679219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:48.758 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:48.756167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:48.770 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:48.769071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:58.841 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:58.839008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:48:58.889 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:48:58.887517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:09.007 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:09.005161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:09.019 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:09.017578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:19.094 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:19.092326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:19.107 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:19.105641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:29.206 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:29.204658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:29.218 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:29.216476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:39.321 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:39.319347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:39.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:39.331009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:49.426 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:49.424233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:49.436 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:49.434063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:59.521 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:59.518864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:49:59.533 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:49:59.531765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:09.601 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:09.599357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:09.613 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:09.611397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:19.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:19.686096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:19.746 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:19.744268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:29.818 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:29.816678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:29.870 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:29.868864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:39.948 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:39.946253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:39.962 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:39.960807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:50.036 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:50.034638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:50:50.092 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:50:50.090902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:00.182 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:00.180416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:00.197 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:00.196174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:10.271 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:10.269111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:10.283 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:10.280999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:20.354 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:20.352240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:20.367 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:20.365647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:30.439 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:30.436924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:30.451 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:30.449753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:40.521 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:40.518962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:40.533 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:40.531405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:50.635 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:50.633717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:51:50.646 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:51:50.644177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:00.758 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:00.756600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:00.768 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:00.766236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:10.835 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:10.832608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:10.847 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:10.845800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:20.953 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:20.951352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:20.966 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:20.964534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:31.096 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:31.094214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:31.107 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:31.105300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:41.178 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:41.175909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:41.190 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:41.188186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:51.306 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:51.303948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:52:51.334 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:52:51.329689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:01.470 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:01.467844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:01.482 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:01.481008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:11.585 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:11.582591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:11.602 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:11.599432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:21.697 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:21.694776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:21.710 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:21.708525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:31.834 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:31.831685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:31.847 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:31.845546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:41.974 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:41.972141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:41.986 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:41.984425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:52.048 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:52.042351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:53:52.059 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:53:52.057368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:02.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:02.193124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:02.207 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:02.203370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:12.318 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:12.315880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:12.331 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:12.329887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:22.398 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:22.396427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:22.408 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:22.406933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:32.523 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:32.520801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:32.541 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:32.539328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:42.615 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:42.613374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:42.633 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:42.631066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:52.737 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:52.735550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:54:52.750 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:54:52.748851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:02.857 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:02.854767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:02.869 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:02.867013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:12.937 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:12.935389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:12.950 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:12.948215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:23.068 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:23.066330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:23.083 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:23.081868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:33.187 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:33.184554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:33.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:33.197890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:43.300 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:43.298237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:43.312 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:43.310193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:53.435 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:53.433411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:55:53.448 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:55:53.446069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:03.544 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:03.541564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:03.557 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:03.555300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:13.636 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:13.634249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:13.649 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:13.646893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:23.716 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:23.714401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:23.729 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:23.727353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:33.869 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:33.866749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:33.881 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:33.879365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:43.989 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:43.986671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:44.002 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:44.000233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:54.140 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:54.138042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:56:54.155 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:56:54.153555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:04.246 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:04.244237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:04.259 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:04.257190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:14.354 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:14.351722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:14.366 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:14.364342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:24.448 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:24.445894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:24.459 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:24.456926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:34.545 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:34.543008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:34.560 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:34.558333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:44.692 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:44.689822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:44.723 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:44.721129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:54.853 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:54.851073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:57:54.865 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:57:54.863223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:04.970 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:04.968355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:04.981 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:04.979200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:15.111 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:15.109338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:15.124 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:15.122033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:25.217 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:25.214838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:25.230 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:25.227950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:35.321 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:35.317648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:35.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:35.331391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:45.454 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:45.452390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:45.468 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:45.466157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:55.587 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:55.584558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:58:55.600 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:58:55.598240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:05.712 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:05.710211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:05.725 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:05.723547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:15.798 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:15.795757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:15.811 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:15.809291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:25.894 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:25.891704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:25.905 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:25.903464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:35.980 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:35.978070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:36.034 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:36.032339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:46.116 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:46.113959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:46.128 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:46.126628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:56.232 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:56.229755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T16:59:56.245 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 16:59:56.242927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:06.357 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:06.354609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:06.372 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:06.369799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:16.496 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:16.494309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:16.511 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:16.508926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:26.586 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:26.583912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:26.634 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:26.632075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:36.712 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:36.710130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:36.800 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:36.797805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:46.896 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:46.893184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:46.907 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:46.904869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:57.029 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:57.026996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:00:57.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:00:57.037638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:07.336 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:07.334137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:07.349 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:07.347250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:17.551 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:17.548955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:17.568 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:17.565890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:27.735 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:27.732828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:27.747 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:27.745249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:30.208 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:30.205688 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T17:01:30.909 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:30.907018 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T17:01:30.894+0000 7fc2f14c7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T17:01:30.910 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T17:01:30.894+0000 7fc2f14c7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T17:01:30.910 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T17:01:30.894+0000 7fc2f14c7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T17:01:30.910 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T17:01:37.815 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:37.813149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:37.829 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:37.826693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:47.924 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:47.921607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:47.936 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:47.933960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:58.005 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:58.002806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:01:58.017 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:01:58.015434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:08.139 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:08.136906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:08.153 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:08.150861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:18.227 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:18.224601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:18.240 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:18.238332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:28.335 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:28.332781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:28.350 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:28.347987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:38.441 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:38.438733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:38.453 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:38.451472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:48.561 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:48.554883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:48.570 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:48.568533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:58.681 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:58.678693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:02:58.696 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:02:58.693876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:08.803 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:08.801098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:08.818 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:08.816383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:18.913 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:18.910730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:18.932 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:18.930110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:29.015 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:29.013445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:29.027 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:29.025062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:39.124 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:39.121578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:39.135 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:39.133227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:49.269 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:49.266577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:49.279 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:49.277147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:59.374 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:59.371735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:03:59.384 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:03:59.382228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:09.538 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:09.535793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:09.549 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:09.547371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:19.649 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:19.646528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:19.662 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:19.660314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:29.776 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:29.773566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:29.790 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:29.787965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:39.913 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:39.910582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:39.926 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:39.924284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:50.015 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:50.013295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:04:50.030 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:04:50.027939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:00.158 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:00.154842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:00.168 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:00.166387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:10.243 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:10.241287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:10.311 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:10.308517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:20.385 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:20.379908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:20.396 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:20.393954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:30.467 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:30.463943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:30.503 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:30.500960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:40.579 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:40.577016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:40.591 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:40.589676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:50.682 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:50.680225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:05:50.698 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:05:50.696417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:00.799 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:00.796387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:00.814 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:00.811748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:10.935 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:10.932808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:10.950 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:10.948046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:21.051 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:21.048452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:21.064 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:21.062574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:31.175 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:31.172879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:31.186 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:31.184023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:41.296 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:41.294283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:41.307 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:41.305548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:51.436 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:51.433034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:06:51.446 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:06:51.444428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:01.550 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:01.547453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:01.570 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:01.568011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:11.644 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:11.641895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:11.657 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:11.655172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:21.762 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:21.758951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:21.777 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:21.773586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:31.896 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:31.894079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:31.908 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:31.906307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:41.982 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:41.979746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:42.049 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:42.046384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:52.126 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:52.124523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:07:52.139 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:07:52.137119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:02.251 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:02.248262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:02.265 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:02.262189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:12.411 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:12.408855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:12.425 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:12.422848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:22.498 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:22.495906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:22.510 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:22.508261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:32.618 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:32.615980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:32.635 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:32.633510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:42.742 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:42.739358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:42.757 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:42.754881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:52.818 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:52.815389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:08:52.832 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:08:52.829722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:02.936 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:02.933187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:02.947 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:02.944506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:13.063 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:13.060219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:13.075 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:13.072678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:23.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:23.178635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:23.193 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:23.190891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:33.286 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:33.283403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:33.299 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:33.295806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:43.406 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:43.403854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:43.421 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:43.419180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:53.554 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:53.550766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:09:53.567 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:09:53.564955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:03.635 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:03.632697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:03.649 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:03.646393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:13.756 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:13.753992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:13.771 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:13.768435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:23.874 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:23.871343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:23.887 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:23.884606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:33.957 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:33.954602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:33.971 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:33.969062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:44.044 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:44.042098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:44.101 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:44.098376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:54.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:54.197356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:10:54.211 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:10:54.208956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:04.323 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:04.319732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:04.347 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:04.344312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:14.440 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:14.437178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:14.458 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:14.456050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:24.538 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:24.533225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:24.545 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:24.543171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:34.614 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:34.612244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:34.631 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:34.628656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:44.744 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:44.741161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:44.756 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:44.754479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:54.819 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:54.816431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:11:54.835 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:11:54.831307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:04.950 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:04.943902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:04.958 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:04.956265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:15.092 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:15.089223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:15.105 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:15.103236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:25.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:25.206914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:25.221 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:25.219310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:35.347 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:35.344857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:35.359 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:35.357056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:45.458 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:45.455479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:45.470 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:45.467783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:55.543 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:55.540668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:12:55.556 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:12:55.553638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:05.660 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:05.657875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:05.672 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:05.669701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:15.787 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:15.784299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:15.799 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:15.796526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:25.861 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:25.857584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:25.903 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:25.900472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:36.030 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:36.027287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:36.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:36.039331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:46.140 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:46.138170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:46.153 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:46.150963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:56.278 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:56.274726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:13:56.286 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:13:56.284052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:06.390 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:06.387667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:06.404 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:06.402210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:16.526 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:16.523164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:16.538 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:16.535614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:26.672 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:26.669231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:26.682 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:26.679843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:36.825 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:36.822550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:36.838 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:36.835947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:46.945 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:46.942423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:46.964 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:46.956571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:57.072 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:57.069792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:14:57.087 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:14:57.084811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:07.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:07.180386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:07.194 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:07.191883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:17.325 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:17.322492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:17.339 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:17.336774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:27.440 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:27.437851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:27.453 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:27.450605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:37.564 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:37.561426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:37.577 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:37.575035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:47.645 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:47.643102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:47.702 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:47.699331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:57.765 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:57.761581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:15:57.853 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:15:57.850686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:07.928 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:07.925887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:07.941 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:07.938539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:18.013 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:18.010542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:18.027 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:18.024716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:28.092 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:28.089296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:28.104 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:28.102110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:38.220 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:38.217553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:38.235 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:38.232833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:48.339 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:48.336437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:48.351 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:48.348955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:58.440 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:58.438034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:16:58.452 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:16:58.449868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:08.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:08.556371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:08.574 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:08.571591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:18.652 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:18.649944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:18.710 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:18.708045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:28.787 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:28.784971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:28.834 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:28.832066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:38.936 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:38.933505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:38.947 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:38.945147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:49.069 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:49.066772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:49.081 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:49.079333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:59.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:59.180992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:17:59.196 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:17:59.193850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:09.262 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:09.259681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:09.274 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:09.272378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:10.916 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:10.913881 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T17:18:11.655 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:11.652185 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T17:18:11.640+0000 7f8d1fd9f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T17:18:11.655 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T17:18:11.640+0000 7f8d1fd9f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T17:18:11.655 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T17:18:11.640+0000 7f8d1fd9f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T17:18:11.656 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T17:18:19.346 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:19.343435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:19.356 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:19.352702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:29.450 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:29.448174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:29.465 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:29.463144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:39.583 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:39.580657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:39.597 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:39.594696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:49.731 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:49.728590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:49.746 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:49.743940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:59.815 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:59.812498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:18:59.829 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:18:59.826587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:09.912 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:09.909802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:09.925 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:09.922915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:19.999 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:19.996822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:20.011 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:20.009358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:30.085 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:30.081880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:30.095 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:30.092505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:40.167 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:40.165031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:40.178 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:40.176467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:50.282 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:50.279456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:19:50.295 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:19:50.291996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:00.376 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:00.373452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:00.387 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:00.385486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:10.483 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:10.480844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:10.495 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:10.492865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:20.570 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:20.567835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:20.583 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:20.580821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:30.679 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:30.676479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:30.692 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:30.690248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:40.832 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:40.830353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:40.849 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:40.846517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:50.952 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:50.949530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:20:50.966 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:20:50.964221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:01.037 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:01.034251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:01.050 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:01.048287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:11.165 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:11.162549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:11.178 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:11.176327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:21.375 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:21.372458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:21.384 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:21.382582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:31.521 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:31.519086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:31.534 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:31.532352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:41.666 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:41.663750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:41.681 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:41.678542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:51.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:51.751703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:21:51.766 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:21:51.764460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:01.862 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:01.859815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:01.873 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:01.871257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:11.972 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:11.969476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:11.985 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:11.982935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:22.080 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:22.078104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:22.094 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:22.092711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:32.161 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:32.158275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:32.172 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:32.170268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:42.243 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:42.241255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:42.256 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:42.254045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:52.372 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:52.369406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:22:52.383 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:22:52.381541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:02.459 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:02.456495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:02.505 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:02.502543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:12.607 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:12.604275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:12.621 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:12.619328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:22.703 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:22.701094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:22.715 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:22.713420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:32.816 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:32.813807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:32.834 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:32.831878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:42.982 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:42.980077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:42.995 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:42.993147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:53.121 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:53.118568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:23:53.133 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:23:53.131408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:03.209 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:03.206187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:03.220 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:03.218261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:13.337 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:13.334364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:13.350 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:13.348488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:23.471 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:23.468507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:23.487 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:23.484861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:33.583 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:33.580825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:33.595 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:33.593519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:43.673 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:43.670937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:43.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:43.686042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:53.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:53.766663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:24:53.779 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:24:53.777363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:03.849 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:03.846486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:03.863 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:03.861016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:13.969 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:13.967264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:13.983 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:13.980844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:24.098 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:24.096213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:24.111 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:24.109202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:34.206 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:34.203790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:34.218 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:34.216483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:44.290 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:44.288071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:44.301 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:44.299726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:54.408 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:54.406080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:25:54.425 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:25:54.421433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:04.521 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:04.518854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:04.534 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:04.532825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:14.616 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:14.614300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:14.630 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:14.628301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:24.711 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:24.708957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:24.763 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:24.760863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:34.837 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:34.834814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:34.854 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:34.852622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:44.959 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:44.956946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:44.971 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:44.969231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:55.076 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:55.074108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:26:55.090 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:26:55.088001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:05.165 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:05.162848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:05.180 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:05.178191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:15.255 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:15.253125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:15.267 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:15.265443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:25.340 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:25.337779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:25.375 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:25.372408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:35.452 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:35.450264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:35.467 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:35.465079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:45.543 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:45.540414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:45.602 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:45.599506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:55.682 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:55.679785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:27:55.695 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:27:55.693538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:05.786 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:05.783636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:05.799 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:05.797345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:15.940 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:15.937696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:15.951 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:15.949247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:26.077 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:26.075113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:26.088 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:26.085888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:36.240 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:36.237037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:36.252 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:36.250114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:46.400 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:46.398063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:46.413 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:46.411310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:56.541 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:56.538848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:28:56.605 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:28:56.550327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:06.681 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:06.678778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:06.693 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:06.691116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:16.819 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:16.816513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:16.831 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:16.829236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:26.956 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:26.954364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:26.972 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:26.970082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:37.085 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:37.082930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:37.097 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:37.095502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:47.165 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:47.163344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:47.179 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:47.177115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:57.295 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:57.293230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:29:57.309 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:29:57.306919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:07.374 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:07.372048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:07.435 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:07.429379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:17.517 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:17.514371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:17.529 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:17.525844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:27.613 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:27.611147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:27.630 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:27.628635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:37.771 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:37.769133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:37.785 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:37.783102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:47.860 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:47.858247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:47.873 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:47.871137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:57.961 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:57.959372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:30:57.976 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:30:57.973757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:08.116 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:08.114333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:08.130 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:08.127918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:18.253 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:18.250860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:18.266 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:18.264168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:28.364 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:28.361671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:28.377 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:28.375565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:38.448 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:38.446634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:38.529 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:38.527481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:48.604 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:48.601692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:48.617 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:48.615525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:58.738 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:58.736298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:31:58.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:31:58.751697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:08.822 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:08.816554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:08.835 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:08.833008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:18.947 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:18.944935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:18.961 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:18.958470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:29.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:29.021773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:29.037 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:29.034981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:39.107 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:39.104786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:39.123 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:39.119285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:49.192 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:49.190017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:49.220 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:49.217751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:59.284 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:59.281483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:32:59.297 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:32:59.294756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:09.401 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:09.399396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:09.415 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:09.413239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:19.513 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:19.510914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:19.528 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:19.526132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:29.633 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:29.630872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:29.650 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:29.647989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:39.722 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:39.720586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:39.733 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:39.731471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:49.858 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:49.856321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:49.867 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:49.865762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:59.980 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:59.977874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:33:59.992 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:33:59.990881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:10.094 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:10.091671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:10.106 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:10.104590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:20.348 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:20.345954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:20.361 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:20.359248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:30.514 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:30.511786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:30.527 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:30.525217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:40.656 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:40.654068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:40.671 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:40.668933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:50.743 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:50.740609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:50.800 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:50.798045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:34:51.662 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:51.659733 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T17:34:52.386 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:34:52.381037 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T17:34:52.371+0000 7f7adbfac380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T17:34:52.386 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T17:34:52.371+0000 7f7adbfac380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T17:34:52.387 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T17:34:52.371+0000 7f7adbfac380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T17:34:52.387 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T17:35:00.928 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:00.926519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:00.943 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:00.940223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:11.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:11.018894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:11.034 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:11.032482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:21.142 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:21.139573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:21.152 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:21.149992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:31.272 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:31.270221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:31.284 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:31.282022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:41.360 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:41.357802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:41.425 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:41.423331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:51.548 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:51.546525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:35:51.562 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:35:51.560214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:01.668 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:01.666291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:01.683 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:01.681622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:11.756 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:11.753573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:11.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:11.766913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:21.874 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:21.872264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:21.889 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:21.880946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:31.993 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:31.991023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:32.008 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:32.006390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:42.121 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:42.119165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:42.134 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:42.132320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:52.232 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:52.230181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:36:52.246 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:36:52.243498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:02.324 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:02.322310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:02.394 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:02.392615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:12.509 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:12.507361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:12.523 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:12.521723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:22.640 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:22.637735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:22.652 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:22.650269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:32.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:32.767375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:32.785 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:32.783090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:42.888 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:42.886575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:42.900 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:42.898602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:52.969 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:52.964845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:37:52.980 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:37:52.978137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:03.044 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:03.041949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:03.056 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:03.054586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:13.192 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:13.190182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:13.207 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:13.205207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:23.296 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:23.294425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:23.310 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:23.308658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:33.392 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:33.390227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:33.403 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:33.401585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:43.477 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:43.475255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:43.545 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:43.543277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:53.617 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:53.615414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:38:53.631 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:38:53.629240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:03.757 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:03.755066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:03.771 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:03.769593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:13.848 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:13.846119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:13.861 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:13.859525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:23.967 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:23.965356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:23.980 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:23.977997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:34.051 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:34.049280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:34.064 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:34.062948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:44.171 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:44.168933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:44.188 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:44.185742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:54.253 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:54.251280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:39:54.267 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:39:54.265206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:04.342 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:04.340379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:04.356 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:04.354035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:14.466 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:14.464467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:14.480 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:14.478486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:24.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:24.556842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:24.569 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:24.567394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:34.669 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:34.667013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:34.683 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:34.681274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:44.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:44.752482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:44.828 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:44.826673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:54.921 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:54.918632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:40:54.935 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:40:54.933404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:05.133 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:05.131249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:05.144 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:05.142519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:15.226 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:15.223726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:15.305 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:15.303634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:25.435 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:25.432608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:25.449 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:25.447603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:35.517 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:35.515383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:35.531 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:35.529640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:45.623 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:45.620799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:45.636 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:45.634044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:55.740 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:55.738175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:41:55.752 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:41:55.750630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:05.832 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:05.829941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:05.843 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:05.841559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:15.918 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:15.916746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:15.930 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:15.928822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:25.995 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:25.992797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:26.006 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:26.004753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:36.134 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:36.132048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:36.144 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:36.142944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:46.278 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:46.275624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:46.291 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:46.289341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:56.393 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:56.390995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:42:56.407 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:42:56.405919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:06.492 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:06.487722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:06.503 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:06.501582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:16.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:16.572985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:16.589 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:16.586937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:26.676 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:26.674204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:26.722 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:26.720436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:36.782 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:36.779105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:36.842 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:36.840167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:46.948 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:46.945948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:46.961 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:46.959162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:57.071 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:57.069698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:43:57.089 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:43:57.087103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:07.155 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:07.153204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:07.168 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:07.166880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:17.287 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:17.284526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:17.308 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:17.299144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:27.379 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:27.376969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:27.392 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:27.390137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:37.495 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:37.492065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:37.511 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:37.509289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:47.602 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:47.600388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:47.617 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:47.614598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:57.699 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:57.696721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:44:57.781 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:44:57.779559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:07.862 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:07.859861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:07.874 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:07.872544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:18.007 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:18.004593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:18.018 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:18.015819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:28.108 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:28.106291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:28.121 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:28.119909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:38.260 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:38.258461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:38.275 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:38.273208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:48.346 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:48.343908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:48.359 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:48.357560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:58.432 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:58.429467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:45:58.480 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:45:58.477892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:08.593 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:08.591625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:08.630 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:08.601810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:18.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:18.706029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:18.721 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:18.719237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:28.784 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:28.782454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:28.798 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:28.795734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:38.897 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:38.894430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:38.909 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:38.907314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:48.977 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:48.974718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:49.035 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:49.033068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:59.105 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:59.103513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:46:59.121 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:46:59.118698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:09.188 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:09.185787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:09.203 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:09.201041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:19.303 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:19.300959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:19.314 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:19.312059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:29.400 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:29.397886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:29.415 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:29.411588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:39.489 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:39.486696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:39.500 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:39.498367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:49.582 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:49.580012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:49.594 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:49.591858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:59.712 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:59.709922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:47:59.727 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:47:59.724744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:09.955 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:09.952668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:09.963 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:09.961911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:20.142 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:20.140190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:20.156 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:20.153781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:30.227 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:30.225018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:30.240 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:30.237914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:40.346 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:40.343728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:40.358 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:40.355873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:50.428 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:50.425867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:48:50.444 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:48:50.441928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:00.512 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:00.510409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:00.528 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:00.527062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:10.661 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:10.659660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:10.674 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:10.672448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:20.796 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:20.794427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:20.808 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:20.806240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:30.870 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:30.868079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:30.882 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:30.880372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:40.998 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:40.996162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:41.015 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:41.012682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:51.101 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:51.099270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:49:51.114 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:49:51.112544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:01.215 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:01.213171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:01.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:01.226812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:11.296 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:11.294673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:11.311 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:11.308818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:21.418 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:21.415225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:21.433 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:21.431051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:31.536 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:31.533941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:31.550 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:31.547568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:41.612 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:41.610521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:41.626 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:41.624804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:51.690 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:51.688477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:50:51.705 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:50:51.701273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:01.800 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:01.798547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:01.815 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:01.812934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:11.933 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:11.930587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:11.945 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:11.942905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:22.020 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:22.018229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:22.096 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:22.093900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:32.225 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:32.222305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:32.236 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:32.234019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:32.415 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:32.413304 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T17:51:33.073 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:33.070572 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T17:51:33.058+0000 7f3764208380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T17:51:33.073 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T17:51:33.058+0000 7f3764208380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T17:51:33.073 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T17:51:33.058+0000 7f3764208380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T17:51:33.074 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T17:51:42.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:42.331058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:42.346 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:42.344233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:52.420 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:52.418330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:51:52.490 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:51:52.488607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:02.593 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:02.590428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:02.607 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:02.605804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:12.732 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:12.729841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:12.746 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:12.744464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:22.875 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:22.872972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:22.890 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:22.888415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:32.995 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:32.993226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:33.010 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:33.008037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:43.083 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:43.081292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:43.098 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:43.096844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:53.211 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:53.209341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:52:53.223 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:52:53.221478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:03.280 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:03.277853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:03.293 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:03.291130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:13.410 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:13.407379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:13.424 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:13.420913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:23.535 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:23.532959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:23.548 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:23.546260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:33.627 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:33.624806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:33.640 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:33.638494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:43.719 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:43.716778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:43.734 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:43.731785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:53.797 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:53.795035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:53:53.810 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:53:53.808659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:03.880 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:03.878165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:03.891 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:03.889703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:14.000 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:13.997824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:14.011 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:14.009054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:24.129 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:24.126947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:24.139 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:24.137889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:34.245 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:34.243333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:34.262 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:34.259911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:44.334 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:44.332266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:44.347 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:44.345312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:54.419 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:54.415266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:54:54.431 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:54:54.428430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:04.530 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:04.528270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:04.543 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:04.541093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:14.642 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:14.640547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:14.655 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:14.652915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:24.748 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:24.746006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:24.761 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:24.758890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:34.839 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:34.835202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:34.851 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:34.848458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:44.925 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:44.919197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:44.936 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:44.933947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:54.997 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:54.994953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:55:55.011 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:55:55.009654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:05.088 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:05.086656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:05.111 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:05.109346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:15.181 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:15.179083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:15.194 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:15.192660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:25.266 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:25.263732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:25.280 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:25.278282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:35.408 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:35.406001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:35.422 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:35.419558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:45.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:45.556426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:45.573 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:45.571065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:55.644 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:55.638042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:56:55.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:56:55.706526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:05.830 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:05.827951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:05.844 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:05.842523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:15.968 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:15.966325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:15.980 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:15.978534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:26.096 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:26.093794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:26.110 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:26.108966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:36.200 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:36.198343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:36.213 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:36.211013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:46.285 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:46.283674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:46.298 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:46.295763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:56.378 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:56.375266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:57:56.426 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:57:56.423895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:06.550 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:06.548587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:06.564 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:06.562620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:16.680 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:16.678233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:16.693 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:16.691344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:26.796 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:26.794183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:26.809 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:26.807059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:36.910 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:36.908210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:36.923 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:36.921124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:47.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:47.022168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:47.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:47.036692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:57.128 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:57.126501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:58:57.143 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:58:57.140706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:07.251 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:07.249212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:07.264 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:07.262971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:17.366 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:17.363395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:17.380 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:17.378028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:27.493 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:27.490686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:27.505 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:27.502882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:37.583 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:37.581240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:37.595 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:37.593006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:47.698 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:47.695865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:47.711 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:47.709265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:57.886 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:57.884208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T17:59:57.898 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 17:59:57.896623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:08.025 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:08.022470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:08.039 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:08.037174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:18.134 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:18.131664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:18.147 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:18.145350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:28.249 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:28.246651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:28.265 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:28.261439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:38.330 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:38.327771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:38.342 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:38.340341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:48.458 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:48.455791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:48.472 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:48.469900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:58.585 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:58.582898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:00:58.595 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:00:58.593411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:08.698 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:08.696004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:08.714 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:08.712183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:18.777 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:18.775410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:18.791 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:18.789023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:28.876 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:28.874505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:28.890 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:28.888770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:39.054 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:39.052385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:39.065 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:39.063896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:49.166 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:49.163270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:49.178 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:49.176194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:59.296 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:59.293910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:01:59.311 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:01:59.308328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:09.369 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:09.367461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:09.382 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:09.380535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:19.544 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:19.541929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:19.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:19.556938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:29.661 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:29.658520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:29.672 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:29.670615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:39.781 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:39.779257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:39.795 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:39.793178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:49.895 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:49.893277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:49.907 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:49.905059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:59.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:59.975562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:02:59.990 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:02:59.988168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:10.057 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:10.054425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:10.068 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:10.066314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:20.130 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:20.128483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:20.148 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:20.145873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:30.236 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:30.234179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:30.295 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:30.293382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:40.364 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:40.362317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:40.373 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:40.371858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:50.449 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:50.447708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:03:50.463 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:03:50.461078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:00.636 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:00.633999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:00.648 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:00.646276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:10.763 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:10.760969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:10.778 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:10.776334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:20.847 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:20.845117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:20.913 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:20.911243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:31.025 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:31.023317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:31.037 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:31.035269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:41.111 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:41.109046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:41.125 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:41.122986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:51.196 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:51.194307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:04:51.210 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:04:51.208203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:01.299 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:01.296025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:01.313 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:01.311212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:11.418 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:11.416231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:11.431 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:11.429064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:21.552 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:21.550069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:21.565 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:21.563920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:31.687 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:31.684876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:31.701 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:31.699277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:41.803 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:41.800794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:41.815 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:41.813649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:51.937 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:51.934976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:05:51.948 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:05:51.946347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:02.023 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:02.021145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:02.041 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:02.039018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:12.158 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:12.156567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:12.170 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:12.168851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:22.278 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:22.276114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:22.304 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:22.290064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:32.375 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:32.372703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:32.386 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:32.384524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:42.524 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:42.522500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:42.536 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:42.534905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:52.623 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:52.621684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:06:52.642 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:06:52.640476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:02.760 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:02.757393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:02.768 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:02.766705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:12.879 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:12.877532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:12.891 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:12.889451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:22.985 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:22.982694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:22.999 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:22.997207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:33.125 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:33.122627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:33.139 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:33.137132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:43.222 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:43.220463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:43.234 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:43.231903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:53.316 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:53.314135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:07:53.346 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:07:53.344209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:03.444 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:03.441901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:03.456 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:03.454157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:13.080 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:13.077707 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T18:08:13.533 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:13.531644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:13.603 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:13.601108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:13.753 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:13.750311 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T18:08:13.736+0000 7fd5a3751380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T18:08:13.753 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T18:08:13.736+0000 7fd5a3751380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T18:08:13.753 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T18:08:13.736+0000 7fd5a3751380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T18:08:13.754 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T18:08:23.691 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:23.689662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:23.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:23.704840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:33.772 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:33.770156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:33.785 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:33.783708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:43.849 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:43.847562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:43.862 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:43.860560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:53.952 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:53.947973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:08:53.965 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:08:53.962626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:04.059 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:04.056985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:04.072 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:04.070371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:14.183 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:14.180372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:14.195 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:14.193681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:24.261 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:24.258845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:24.323 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:24.321535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:34.398 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:34.396748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:34.412 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:34.410644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:44.506 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:44.503846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:44.519 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:44.517775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:54.618 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:54.616281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:09:54.691 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:09:54.689096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:04.780 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:04.777951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:04.792 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:04.790126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:14.889 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:14.887403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:14.903 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:14.901356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:24.973 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:24.970823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:24.987 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:24.985582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:35.092 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:35.087768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:35.102 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:35.099670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:45.208 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:45.206665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:45.221 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:45.219142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:55.291 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:55.288976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:10:55.306 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:10:55.302460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:05.413 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:05.410815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:05.425 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:05.422917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:15.503 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:15.500974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:15.526 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:15.524506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:25.640 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:25.637161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:25.655 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:25.650328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:35.720 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:35.716420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:35.733 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:35.731013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:45.805 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:45.803101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:45.855 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:45.852938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:55.922 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:55.919650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:11:55.934 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:11:55.932383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:06.058 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:06.056036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:06.071 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:06.068896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:16.173 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:16.171031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:16.191 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:16.188049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:26.299 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:26.297136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:26.312 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:26.310952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:36.389 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:36.387367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:36.401 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:36.399484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:46.478 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:46.476441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:46.491 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:46.488173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:56.564 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:56.561724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:12:56.578 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:12:56.576983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:06.650 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:06.647914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:06.662 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:06.660382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:16.734 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:16.732417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:16.748 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:16.746296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:26.823 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:26.821222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:26.901 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:26.898526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:36.994 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:36.992587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:37.009 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:37.007388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:47.133 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:47.131453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:47.146 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:47.144493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:57.247 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:57.245844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:13:57.261 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:13:57.259780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:07.333 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:07.331846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:07.387 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:07.384862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:17.485 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:17.483076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:17.496 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:17.494988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:27.586 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:27.584213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:27.601 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:27.597713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:37.679 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:37.674639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:37.693 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:37.689013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:47.766 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:47.764614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:47.783 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:47.780447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:57.852 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:57.850319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:14:57.869 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:14:57.866855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:07.966 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:07.963830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:07.978 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:07.975620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:18.109 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:18.106912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:18.121 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:18.119903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:28.214 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:28.212148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:28.227 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:28.225350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:38.329 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:38.327310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:38.341 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:38.339406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:48.409 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:48.407390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:48.425 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:48.423991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:58.526 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:58.522889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:15:58.538 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:15:58.536944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:08.663 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:08.661196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:08.673 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:08.672214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:18.754 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:18.752298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:18.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:18.767643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:28.895 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:28.893200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:28.908 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:28.905521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:38.989 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:38.987017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:38.998 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:38.996463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:49.097 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:49.095255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:49.110 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:49.108797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:59.240 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:59.238135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:16:59.254 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:16:59.252583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:09.382 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:09.379421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:09.396 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:09.393825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:19.506 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:19.504691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:19.520 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:19.518419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:29.594 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:29.592822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:29.651 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:29.649808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:39.771 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:39.769575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:39.787 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:39.785894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:49.864 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:49.862728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:49.877 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:49.874772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:17:59.994 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:17:59.992047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:00.007 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:00.005850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:10.157 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:10.155735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:10.169 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:10.167492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:20.244 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:20.242511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:20.259 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:20.257845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:30.339 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:30.337246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:30.352 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:30.351086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:40.468 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:40.466265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:40.481 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:40.479129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:50.559 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:50.557107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:18:50.580 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:18:50.579087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:00.698 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:00.696589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:00.712 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:00.710859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:10.794 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:10.792880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:10.825 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:10.823787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:20.898 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:20.895925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:20.933 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:20.930632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:30.995 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:30.993468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:31.052 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:31.050692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:41.178 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:41.176589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:41.192 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:41.191270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:51.308 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:51.305679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:19:51.319 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:19:51.317835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:01.466 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:01.464013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:01.479 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:01.476667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:11.580 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:11.578469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:11.592 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:11.590394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:21.712 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:21.710459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:21.725 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:21.723605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:31.848 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:31.846559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:31.874 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:31.869770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:41.943 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:41.941405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:41.957 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:41.955746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:52.082 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:52.080312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:20:52.093 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:20:52.090981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:02.218 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:02.216533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:02.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:02.229677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:12.332 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:12.330386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:12.345 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:12.343865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:22.404 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:22.401707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:22.416 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:22.415286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:32.567 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:32.559975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:32.579 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:32.577986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:42.644 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:42.642539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:42.711 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:42.709016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:52.790 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:52.787947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:21:52.801 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:21:52.799777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:02.859 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:02.853743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:02.915 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:02.913926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:12.992 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:12.990770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:13.003 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:13.001979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:23.163 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:23.161685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:23.178 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:23.176281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:33.260 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:33.259030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:33.274 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:33.272285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:43.371 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:43.369771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:43.383 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:43.381908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:53.448 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:53.446477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:22:53.463 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:22:53.461879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:03.576 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:03.573843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:03.588 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:03.586273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:13.702 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:13.700476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:13.716 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:13.714439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:23.830 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:23.828483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:23.845 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:23.843436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:33.942 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:33.940793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:33.955 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:33.953310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:44.022 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:44.019770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:44.034 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:44.032557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:54.147 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:54.145066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:23:54.160 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:23:54.158918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:04.222 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:04.220269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:04.235 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:04.233599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:14.345 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:14.343173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:14.357 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:14.355769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:24.419 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:24.415140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:24.462 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:24.460903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:34.526 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:34.524074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:34.538 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:34.536667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:44.608 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:44.607109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:44.622 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:44.620473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:53.761 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:53.759325 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T18:24:54.485 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:54.483116 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T18:24:54.475+0000 7f737232d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T18:24:54.485 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T18:24:54.475+0000 7f737232d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T18:24:54.486 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T18:24:54.475+0000 7f737232d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T18:24:54.486 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T18:24:54.726 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:54.724979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:24:54.739 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:24:54.737768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:04.848 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:04.845971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:04.861 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:04.859501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:14.995 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:14.993682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:15.008 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:15.006742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:25.075 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:25.073068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:25.140 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:25.138377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:35.212 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:35.210539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:35.225 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:35.223725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:45.350 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:45.348819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:45.364 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:45.362603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:55.476 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:55.474472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:25:55.490 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:25:55.488351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:05.588 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:05.586591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:05.602 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:05.600419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:15.675 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:15.673835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:15.688 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:15.686642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:25.760 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:25.758787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:25.775 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:25.773854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:35.888 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:35.885971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:35.990 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:35.898166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:46.017 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:46.015769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:46.032 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:46.030715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:56.157 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:56.155455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:26:56.170 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:26:56.168715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:06.292 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:06.285538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:06.302 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:06.301120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:16.402 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:16.400050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:16.416 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:16.414657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:26.518 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:26.516598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:26.536 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:26.534328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:36.629 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:36.627962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:36.640 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:36.639126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:46.737 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:46.734895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:46.746 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:46.744938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:56.858 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:56.856545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:27:56.870 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:27:56.868952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:06.985 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:06.983745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:06.997 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:06.995734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:17.105 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:17.103312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:17.118 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:17.116919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:27.188 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:27.186754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:27.199 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:27.198177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:37.307 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:37.305532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:37.319 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:37.317786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:47.396 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:47.394607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:47.462 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:47.460690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:57.583 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:57.581320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:28:57.599 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:28:57.595229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:07.667 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:07.665271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:07.681 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:07.679646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:17.793 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:17.790200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:17.806 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:17.804907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:27.877 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:27.876189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:27.892 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:27.890758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:37.994 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:37.992771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:38.007 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:38.006234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:48.076 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:48.074365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:48.088 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:48.086866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:58.205 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:58.203832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:29:58.218 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:29:58.216678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:08.311 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:08.310016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:08.326 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:08.325195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:18.460 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:18.459174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:18.474 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:18.473415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:28.568 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:28.566516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:28.580 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:28.579499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:38.663 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:38.661276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:38.675 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:38.673604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:48.757 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:48.755568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:48.769 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:48.767851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:58.880 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:58.878535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:30:58.892 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:30:58.891347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:08.984 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:08.982426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:08.997 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:08.996020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:19.063 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:19.061715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:19.075 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:19.074244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:29.142 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:29.140398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:29.157 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:29.155636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:39.274 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:39.272422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:39.286 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:39.284790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:49.407 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:49.405914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:49.437 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:49.435457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:59.510 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:59.508696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:31:59.523 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:31:59.522268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:09.646 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:09.644551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:09.659 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:09.658016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:19.752 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:19.750614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:19.765 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:19.764062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:29.869 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:29.867651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:29.884 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:29.883468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:39.996 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:39.994362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:40.010 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:40.009135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:50.116 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:50.114879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:32:50.130 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:32:50.129374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:00.259 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:00.257219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:00.273 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:00.272458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:10.350 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:10.348729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:10.418 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:10.416471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:20.484 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:20.483307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:20.563 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:20.498647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:30.574 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:30.573131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:30.585 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:30.583677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:40.655 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:40.653964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:40.706 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:40.705359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:50.823 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:50.821718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:33:50.838 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:33:50.836702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:00.941 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:00.939435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:00.952 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:00.951097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:11.026 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:11.025159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:11.040 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:11.039577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:21.120 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:21.118505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:21.164 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:21.163383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:31.232 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:31.231302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:31.297 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:31.295535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:41.419 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:41.416598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:41.431 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:41.428540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:51.607 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:51.605450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:34:51.621 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:34:51.619502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:01.694 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:01.692802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:01.707 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:01.705812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:11.773 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:11.772070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:11.785 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:11.783881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:21.916 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:21.914382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:21.927 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:21.926667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:32.012 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:32.011317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:32.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:32.023455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:42.081 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:42.080146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:42.093 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:42.092135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:52.213 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:52.211756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:35:52.229 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:35:52.227430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:02.334 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:02.332354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:02.347 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:02.345863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:12.445 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:12.444150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:12.459 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:12.457454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:22.585 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:22.583871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:22.597 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:22.596134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:32.667 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:32.666202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:32.729 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:32.728265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:42.803 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:42.802081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:42.817 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:42.815787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:52.919 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:52.917117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:36:52.931 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:36:52.929817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:03.033 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:03.032062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:03.046 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:03.045575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:13.182 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:13.180350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:13.194 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:13.193479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:23.359 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:23.358261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:23.372 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:23.371132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:33.440 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:33.438592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:33.479 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:33.478104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:43.603 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:43.601750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:43.616 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:43.615432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:53.685 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:53.684203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:37:53.700 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:37:53.698839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:03.779 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:03.777429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:03.791 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:03.790269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:13.855 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:13.854345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:13.869 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:13.868442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:23.942 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:23.940452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:23.979 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:23.978497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:34.125 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:34.123801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:34.136 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:34.134869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:44.211 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:44.210057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:44.268 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:44.266773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:54.383 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:54.381363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:38:54.397 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:38:54.396452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:04.546 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:04.545194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:04.557 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:04.556381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:14.623 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:14.621339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:14.693 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:14.692166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:24.818 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:24.816940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:24.831 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:24.830550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:34.890 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:34.889449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:34.903 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:34.902622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:45.026 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:45.025433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:45.038 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:45.037323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:55.155 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:55.153797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:39:55.168 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:39:55.167570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:05.236 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:05.235424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:05.249 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:05.248414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:15.316 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:15.314598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:15.328 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:15.327595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:25.443 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:25.441417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:25.457 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:25.455817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:35.565 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:35.564141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:35.578 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:35.577547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:45.708 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:45.707293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:45.723 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:45.722617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:55.813 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:55.811650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:40:55.826 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:40:55.825530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:05.934 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:05.932770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:05.943 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:05.942111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:16.095 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:16.093737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:16.109 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:16.108302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:26.222 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:26.220590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:26.236 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:26.235471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:34.494 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:34.492449 I | op-mon: parsing mon endpoints: b=10.96.127.40:6789,c=10.111.185.43:6789,a=10.97.100.133:6789 2022-03-21T18:41:35.221 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:35.219569 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-21T18:41:35.209+0000 7f4094f57380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-21T18:41:35.221 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T18:41:35.209+0000 7f4094f57380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-21T18:41:35.222 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21T18:41:35.209+0000 7f4094f57380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-21T18:41:35.222 INFO:tasks.rook.operator.smithi043.stdout:couldn't init storage provider. : exit status 5 2022-03-21T18:41:36.346 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:36.344982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:36.360 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:36.358918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:46.433 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:46.431864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:46.496 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:46.494895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:56.603 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:56.601608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:41:56.615 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:41:56.614716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:06.702 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:06.701109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:06.714 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:06.713545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:16.825 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:16.823168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:16.838 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:16.837307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:26.958 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:26.956760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:26.968 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:26.966919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:37.073 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:37.072057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:37.087 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:37.085590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:47.166 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:47.164707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:47.224 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:47.222718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:57.293 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:57.292236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:42:57.307 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:42:57.306576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:07.432 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:07.430282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:07.445 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:07.443964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:17.578 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:17.576982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:17.591 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:17.589767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:27.667 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:27.665832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:27.680 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:27.678868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:37.799 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:37.797602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:37.813 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:37.812292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:47.933 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:47.932333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:47.947 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:47.946126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:58.016 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:58.014936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:43:58.025 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:43:58.024127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:08.101 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:08.100171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:08.116 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:08.115322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:18.245 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:18.243656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:18.256 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:18.255845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:28.364 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:28.362620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:28.376 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:28.375478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:38.448 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:38.447016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:38.460 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:38.459621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:48.575 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:48.574311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:48.586 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:48.585921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:58.698 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:58.697396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:44:58.717 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:44:58.716363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:08.814 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:08.812931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:08.897 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:08.896676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:19.002 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:19.000635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:19.015 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:19.014091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:29.141 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:29.140366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:29.156 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:29.154822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:39.218 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:39.217516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:39.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:39.229926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:49.323 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:49.321716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:49.337 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:49.335713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:59.440 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:59.439444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:45:59.450 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:45:59.449588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:09.580 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:09.578458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:09.592 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:09.590995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:19.660 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:19.658947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:19.672 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:19.670910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:29.818 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:29.816660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:29.831 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:29.829768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:39.950 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:39.948693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:39.966 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:39.965003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:50.066 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:50.064897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:46:50.079 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:46:50.077890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:00.153 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:00.152308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:00.168 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:00.167210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:10.231 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:10.229610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:10.247 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:10.245960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:20.324 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:20.322846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:20.338 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:20.336990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:30.411 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:30.410404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:30.426 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:30.425297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:40.496 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:40.495171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:40.508 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:40.507883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:50.626 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:50.624375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:47:50.653 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:47:50.651875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:00.729 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:00.728558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:00.742 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:00.741797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:10.811 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:10.810118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:10.891 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:10.889781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:21.024 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:21.022975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:21.037 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:21.036178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:31.112 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:31.110998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:31.125 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:31.124172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:41.198 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:41.196816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:41.210 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:41.209401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:51.329 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:51.327854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:48:51.343 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:48:51.342912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:01.429 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:01.427620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:01.445 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:01.443787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:11.585 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:11.584607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:11.597 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:11.596504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:21.721 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:21.719640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:21.736 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:21.735861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:31.834 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:31.832856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:31.846 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:31.845261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:41.983 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:41.981784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:41.995 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:41.994870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:52.131 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:52.130491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:49:52.145 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:49:52.144520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:02.248 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:02.247262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:02.261 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:02.260736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:12.335 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:12.334307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:12.350 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:12.349473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:22.486 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:22.484999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:22.499 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:22.498739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:32.585 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:32.584019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:32.598 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:32.597455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:42.660 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:42.659057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T18:50:42.676 INFO:tasks.rook.operator.smithi043.stdout:2022-03-21 18:50:42.675300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-21T21:11:41.785 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-21T21:11:41.815 DEBUG:teuthology.task.console_log:Killing console logger for smithi043 2022-03-21T21:11:41.818 DEBUG:teuthology.task.console_log:Killing console logger for smithi065 2022-03-21T21:11:41.819 DEBUG:teuthology.task.console_log:Killing console logger for smithi154 2022-03-21T21:11:41.819 DEBUG:teuthology.exit:Finished running handlers